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Abstract 

Cost  overruns  in  weapon  system  purchases  have  plagued  the  Department  of  Defense 
(DoD)  throughout  its  history  and  have  resulted  in  schedule  delays  and  potentially  reduced 
combat  capability.  This  thesis  created  an  empirical  model  that  begins  to  explain  those  cost 
overruns.  The  model  describes  how  changes  in  defense  budgets,  consolidation  of  the 
defense  industry,  acquisition  refonn,  war,  and  cost  estimating  error  are  related  to  cost 
overruns. 

The  cost  performance  of  186  major  weapon  system  programs  managed  by  the  Air 
Force,  Anny,  and  Navy  from  1970  to  2002  was  described  using  a  panel  regression  model. 
This  research  found  that  funding  instability  resulting  from  changing  levels  of  defense 
budgets  accounted  for  an  increase  of  over  $13.3  billion  in  weapon  system  costs  since 
1970.  This  research  also  found  that  the  defense  industry  consolidation  of  the  1990’s  did 
not  result  in  significant  savings  to  the  DoD.  Finally,  this  research  found  that  contrary  to 
past  studies,  several  acquisition  reforms  are  correlated  with  a  decrease  in  weapon  system 
cost  overruns.  In  particular,  reforms  resulting  from  the  Nunn-McCurdy  Act  of  1982,  the 
Packard  Commission  Recommendations  of  1986  and  the  Federal  Acquisition 
Streamlining  Act  (FAS  A)  of  1994  resulted  in  savings  of  almost  $124  billion  since  1982. 
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THE  IMPACT  OF  ECONOMIC  FACTORS  AND  ACQUISITION  REFORMS  ON  THE 


COST  OF  DEFENSE  WEAPON  SYSTEMS 


I.  Introduction 

The  problem  of  cost  growth  in  weapon  system  development. ..has  been  a  reoccurring  theme  in  acquisition 
reform  for  the  last  several  decades.  Despite  its  high  visibility,  there  has  been  little  systematic  and 
consistent  analysis  of  cost  growth  patterns  and  trends  and  factors  that  effect  cost  growth. 

-Jarvaise  et  al„  1996:iii 


General  Issue 

According  to  a  1993  RAND  study,  the  average  cost  growth  of  a  Department  of 
Defense  (DoD)  Acquisition  Category  I  (ACAT  I)  program  was  20  percent  (Drezner  et  ah, 
1 993  :xiii).  More  recent  research,  such  as  an  October  2000  study  by  the  Defense 
Acquisition  University  (DAU),  has  identified  cost  growth  in  research  and  development 
programs  as  high  as  40  percent  (Swank  et  al,  2000:iii).  Figure  1  depicts  the  percentage  of 
cost  overruns  per  year  on  major  Air  Force,  Army,  and  Navy  programs  from  1970  to  2002. 
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Cost  Overruns  (All  Contracts) 


Air  Force  - • -  Army 

Navy 


Figure  1.  Cost  Overruns  All  Services  (1970-2002) 
(DAES  Database,  Author’s  Calculations:2006) 


To  understand  the  magnitude  of  this  problem,  consider  that  in  2004  the  Air  Force 
alone  had  an  annual  budget  of  over  $  127  billion,  enough  that  if  it  were  a  company  it  would 
be  the  eleventh  largest  in  the  world.  Similarly,  with  a  2004  budget  of  over  $441  billion,  the 
Department  of  Defense  (DoD)  would  be  by  far  the  largest  company  in  the  world  according 
to  Forbes  Magazine’s  list  of  largest  companies  by  total  sales  (Forbes  Magazine,  2004). 
Notice  in  Table  1  that  nearly  a  third  of  the  DoD  budget  in  2004  was  dedicated  to  Production 
and  Research  and  Development  activities. 


Table  1.  2004  DoD  Budget  Summary  (National  Defense  Budget  Estimates,  2006:67) 


2004  DoD  Budget  Summary  (In  billions  of  FY2004  Dollars) 

Appropriation 

Obligation  Authority 

Percentage 

Production  and  R&D 

$158.0 

33% 

Operations  and  Maintenance 
and  Other 

$322.3 

67% 
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Since  defense  budgets  are  essentially  fixed  in  a  given  year,  cost  overruns  create  serious 
problems  for  defense  acquisition  managers  and  for  policy  makers.  They  necessitate 
extensions  of  program  schedules,  and  potentially  cause  a  reduction  in  weapon  system 
quantities  and  capabilities,  as  well  as  funding  instability  in  even  those  programs  that  did  not 
experience  cost  overruns  but  were  affected  by  reprogramming  of  funds.  This  instability  in 
funding  only  further  exacerbates  cost  overruns  (McNutt,  1998:307). 

In  response  to  cost  and  schedule  growth  and  other  problems,  Congress,  the  DoD,  and  the 
individual  military  services  have  instituted  a  series  of  changes  to  the  acquisitions  process 
that  include  changes  to  the  law,  updates  to  military  regulation,  and  an  overhaul  of  the  DoD 
acquisition  training  process.  Table  2  is  a  list  of  major  refonns  to  the  defense  acquisition 
process  in  the  last  20  years.: 

Table  2.  Major  Acquisition  Reforms  (Modified  from  Scofield,  2003:19) 


Major  Acquisition  Reforms  Efforts 

Year 

Effort 

1982 

Nunn-McCurdy  Act 

1986 

Packard  Commission 

1990 

Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA) 

1994 

Federal  Acquisition  Streamlining  Act  (FASA) 

Specific  Issue 

Research  on  DoD  cost  overruns  is  abundant  and  has  ranged  from  descriptive  studies, 
like  the  1993  and  1996  RAND  studies  by  Jarvaise  et  al.  and  Drezner  et  al.,  to  case  studies 
like  Singleton’s  1991  Air  Force  Institute  of  Technology  (AFIT)  thesis  and  McNutt’s  1998 
Massachusetts  Institute  of  Technology  dissertation.  Others  such  as  Searle  (1997), 
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Christensen  et  al.  (1999),  Holbrook  (2003),  and  Phillips  (2004)  have  investigated  the  impact 
of  acquisition  reforms  on  cost  overruns. 

The  whole  of  this  research  has  identified  several  factors  that  may  be  correlated  with  cost 
overruns  yet  in  large  part,  past  research  has  not  been  able  to  empirically  quantify  or  model 
their  relative  impact.  Table  3  is  a  list  of  some  of  the  factors  identified  by  past  research  as 
being  related  to  cost  overruns  and  the  direction  of  impact  suggested  by  the  past  research. 

Table  3.  Factors  Impacting  Cost  Overruns 


Factors 

Expected  Direction  of  Impact 

Acquisition  Reforms 

Inconclusive 

Funding  Instability 

+ 

Estimation  Error 

+ 

Defense  Industry  Consolidation 

- 

War 

+ 

While  past  research  on  cost  overruns  has  been  split  with  some  researchers 
investigating  its  causes  and  others  studying  the  impact  of  acquisition  reform,  this  research 
will  illustrate  that  those  two  areas  of  research  are  really  one  and  that  an  aggregate  model 
of  cost  overruns  that  accounts  for  these  factors  in  a  single  model  is  needed. 

Research  Objectives 

The  motivation  for  this  research  is  the  lack  of  empirical  models  that  explain  the 
causes  of  cost  overruns  in  defense  weapon  system  purchases.  The  purpose  of  this 
research  is  to  create  a  model  of  defense  weapon  system  cost  growth  that  can  be  used  to 
answer  the  following  questions: 
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1.  Did  the  defense  industry  consolidation  and  concurrent  decrease  in  defense 
budgets  that  occurred  in  the  1990’s  affect  the  cost  of  defense  weapon 
systems? 

2.  Is  war  correlated  with  an  increase  in  weapon  system  cost  overruns? 

3.  Is  estimation  error  caused  by  unexpected  inflation  correlated  with  an  increase 
in  weapon  system  cost  overruns? 

4.  Did  acquisition  reforms  have  an  impact  on  cost  overruns  when  defense 
industry  consolidation,  inflation,  changes  in  the  defense  budget,  and  war  are 
considered? 

This  is  not  an  exhaustive  list  of  the  factors  potentially  correlated  with  defense 
weapon  system  cost  overruns.  However  these  are  many  of  the  major  factors  and  by 
modeling  them,  this  study  will  add  greatly  to  the  understanding  of  the  causes  of  defense 
cost  overruns. 

Scope  and  Methodology 

This  research  is  limited  to  the  study  of  contract  cost  overruns  on  Acquisition 
Category  (ACAT)  I  programs  of  the  Anny,  Navy,  and  Air  Force  from  1970  until  2002. 
Defense  contract  cost  overruns  for  each  cross-sectional  unit  are  analyzed  using  a  panel 
regression  model  with  annual  contract  cost  overruns  over  time  as  the  dependant  variable  and 
defense  budgets,  industry  concentration,  inflation,  acquisition  refonns,  and  war  as  the 
independent  variables.  The  panel  regression  model  is  more  robust  than  a  hypothesis  test 
design  or  case  studies  that  are  common  in  past  research  because  the  panel  model  reveals  the 
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relative  importance  of  each  variable  and  its  contribution  to  cost  overruns  in  a  dynamic 
setting.  Consistent  with  past  studies  such  as  Searle  (1997)  and  Holbrook  (2003),  this 
research  investigates  cost  overruns  by  contract  type  and  program  phase.  Table  4  lists 
these  four  models. 


Table  4.  Model  Descriptions 


Model 

Description 

Model  1 

Fixed  Price  Cost  Overruns 

Model  2 

Cost-Plus  Cost  Overruns 

Model  3 

Production  Cost  Overruns 

Model  4 

Research  and  Development  Cost  Overruns 

Organization  of  the  Study 

This  chapter  presented  the  problem  of  cost  overruns  in  defense  weapon  system 
purchases.  Additionally,  this  chapter  identified  the  research  questions  that  will  be 
explored  throughout  this  thesis.  Chapter  II  will  present  a  detailed  discussion  of  past 
research  on  cost  overruns,  followed  by  a  discussion  of  the  data  and  methods  used  in  this 
research  in  Chapter  III,  a  presentation  of  analysis  and  results  in  Chapter  IV,  and  a 
discussion  of  key  findings  in  Chapter  V. 
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II.  Literature  Review 


Chapter  Overview 

This  chapter  reviews  the  previous  research  on  the  subject  of  cost  overruns 
incurred  by  DoD  in  the  acquisition  of  weapon  systems.  Additionally,  this  chapter 
summarizes  the  major  efforts  by  DoD  and  the  United  States  Congress  to  reform  that 
process.  Also,  in  the  years  following  the  various  acquisition  reforms,  much  research  has 
been  conducted  assessing  the  impact  of  those  reforms  on  cost  growth.  This  chapter  will 
review  that  literature  and  summarize  its  major  findings.  Additionally,  while  past  research 
on  cost  growth  has  been  split  with  some  researchers  investigating  its  causes  and  others 
studying  the  impact  of  acquisition  reform,  this  literature  review  will  illustrate  that  those 
two  areas  of  research  are  really  one  and  that  an  aggregate  model  of  cost  growth  that 
accounts  for  these  factors  in  a  single  model  is  needed. 

Past  Research  on  Cost  Growth 

As  implied  above,  much  research  on  cost  overruns  has  been  conducted,  ranging 
from  case  studies  that  investigate  managerial  issues  contributing  to  cost  growth,  to 
descriptive  studies  that  measure  and  characterize  cost  growth.  Table  5  is  a  summary  of 
some  of  this  research.  A  discussion  of  selected  research  follows. 
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Table  5.  Summary  of  Research  on  Weapon  System  Cost  Overruns 


Summary  of  Research  on  Weapon  System  Cost  Overruns 

Author 

Year 

Method 

Main  Findings 

Singleton 

1991 

Case  Study 

Factors  Causing  Cost  Growth:  Funding  Instability, 
Configuration  Instability,  Technology  Readiness 

RAND 

1993,1996 

Descriptive 

20%  Average  Cost  Growth  in  ACAT  1  Programs, 
Program  Maturity  Affects  Cost  Growth 

Czelusniak  and 
Rogers 

1997 

Descriptive 

Funding  Instability  from  Contingency  Operations 
lead  to  cost  growth 

GAO 

1998 

Descriptive 

DoD  expects  significant  savings  from  defense 
industry  consolidation 

McNutt 

1998 

Case  Study 

Complex  Acquisition  Process,  Poor  Portfolio 
Management,  Increased  Cycle  Time,  and  Funding 
Instability  drive  Cost  Growth 

Swank  et  al. 

2000 

Descriptive 

40%  Cost  Growth  in  Research  and  Development 
Programs 

Coleman  et  al 

2003 

Empirical 

No  Relationship  between  Cost  and  Schedule  Growth 

GAO 

2005 

Descriptive 

Consolidation  of  the  Defense  industry  is  at  a  50  year 
high.  DoD  expects  significant  savings  from 
consolidation 

In  her  1991  thesis,  Singleton  attempted  to  predict  “a  range  of  potential  cost 
growth  around  the  most  probable  cost  estimate”  (Singleton,  1991:7).  She  accomplished 
this  goal  by  “researching  the  cost  growth  experienced  in  recent  programs  and 
categorizing  (those)  programs  based  on  several  factors”  (Singleton,  1991:7)  spelled  out  in 
her  thesis.  She  then  identified  the  top  three  contributing  factors  through  a  selection 
process  involving  a  panel  of  cost  analysts.  Her  research  identified  technical  risk, 
configuration  stability,  and  schedule  risk  as  the  top  three  factors  contributing  to  cost 
growth. 

According  to  a  1998  GAO  report  entitled  Defense  Industry:  Consolidation  and 
Options  for  Preserving  Competition,  the  Department  of  Defense  has  identified  12 
industrial  market  sectors  comprised  of  types  of  products  or  weapons  systems  important  to 
U.S.  national  security  interests  (GAO,  1998:2).  The  report  lists  ten  of  those  sectors  and 
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identifies  them  as  experiencing  industry  consolidation  between  1990  and  1998  (GAO, 
1998:10).  The  report  goes  on  to  say  that  “DoD  expects  significant  cost  savings  will  result 


from  (this)  consolidation  (GAO,  1998:2).” 


Table  6.  Reduction  in  the  Number  of  Prime  Contractors 
(GAO,  1998:10) 


Reduction  in  the  Number  of  Prime  Contractors 


Sector  1990 

Tactical  Missiles  13 

Fixed-Wing  Aircraft  8 

Expendable  Launch  Vehicles  6 

Satellites  8 

Surface  Ships  8 

Tactical  Wheeled  Vehicles  6 

Tracked  Combat  Vehicles  3 

Strategic  Missiles  3 

Torpedoes  3 

Rotary-Wing  Aircraft  4 


Note:  The  Electronics  sector  is  not  included. 


1998 

4 

3 
2 

5 
5 

4 
2 
2 
2 
3 


Following  their  1998  report,  the  GAO  issued  another  report  in  2005  called 
“Consolidation  and  Options  for  preserving  Competition”  in  which  they  state  “The  sharp 
decline  in  spending  by  DoD  since  1985  has  resulted  in  a  dramatic  consolidation  of  the 
defense  industry,  which  is  now  more  concentrated  than  at  any  time  in  more  than  a  half 
century.”  (GAO,  2005:  1)  and  then  go  on  to  reiterate  that  the  DoD  expected  significant 
savings  from  consolidation.  (GAO,  2005:2) 

Coleman,  Summerville,  and  Dameron  (2003)  investigated  the  relationship 
between  cost  growth  and  schedule  growth.  They  used  data  from  the  Selected  Acquisition 
Report  (SAR)  compiled  in  the  1993  RAND  Cost  Growth  Database  to  perfomi  this 
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analysis.  The  major  conclusion  of  the  analysis  was  that  there  is  no  correlation  between 
schedule  length  and  cost  growth  (Coleman  et  al,  2003: 120).  One  limitation  of  this 
research  is  that  when  the  requirements  of  a  program  change  significantly,  the  estimate  in 
the  SAR  is  rebaselined.  This  rebaselining  can  obscure  the  actual  cost  and  schedule 
overruns.  If  the  researchers  did  not  account  for  this  technique,  and  there  is  no  evidence 
that  they  did,  then  the  conclusion  that  cost  and  schedule  growth  are  unrelated  is  in  doubt. 

One  common  theme  in  much  of  the  research  on  cost  overruns  is  the  idea  that 
funding  instability  causes  cost  overruns.  As  far  back  as  1991,  Singleton  identified  this  as 
one  of  the  key  contributing  factors.  McNutt  (1998)  argues  among  other  things  that  the 
complex  acquisition  and  budget  process  leads  managers  to  make  suboptimal  funding 
choices  that  lead  to  poor  portfolio  management  and  a  chronic  under- funding  of  long-tenn 
projects  (McNutt,  1998:307).  He  suggests  that  this  instability  in  funding  leads  to 
increased  acquisition  cycle  time  which  in  turn  leads  to  increased  cost  overruns.  In 
support  of  this  idea,  a  quick  glance  at  Figure  2  reveals  a  clear  countercyclical  relationship 
between  cost  overruns  and  changes  in  defense  budgets: 
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Air  Force  Cost  Overruns  and  Budgets 


Year 


-•-Air  Force  Cost  Overruns  -■-%  Change  in  Air  Force  Budget 


Figure  2.  Air  Force  Cost  Overruns  and  Percent  Change  in  Budgets 


This  idea  is  summed  up  well  by  the  US  Senate  in  a  recent  report  on  defense 
acquisition  policy  in  which  they  state:  .  .the  committee  believes  that  one  answer  can  be 
found  in  the  inability  of  the  Department  (of  Defense)  to  address  the  budget  and  program 
instability  issues.  . .  .Funding  and  requirements  instability  continue  to  drive  up  costs  and 
delay  eventual  fielding  of  new  systems”  (Kadish  et  ah,  2005: 1). 

Past  research  also  suggests  that  funding  needs  related  to  contingency  operations 
such  as  Bosnia,  Afghanistan,  and  Iraq  also  contribute  to  cost  growth.  Christensen  et  al. 
(1999)  characterize  this  finding  by  Czelusniak  and  Rogers  (1997)  as  follows: 

“. .  .Czelusniak  and  Rodgers  (1997)  report  that  Congressional  decisions  to  shift  funds  to 
near-term  priorities  external  to  the  program  (e.g.  unplanned  contingency  operations  in 
Bosnia)  account  for  up  to  one-half  of  the  cost  growth  in  major  weapons  systems.” 
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Finally,  the  idea  that  estimation  error  can  lead  to  cost  overruns  is  briefly 
suggested  by  Christensen  et  al.  (1999:1)  and  Drezner  et  al.  ( 1 993 :iii).  Essentially,  the 
idea  is  that  cost  overruns  can  be  caused  by  poor  initial  estimates  of  program  cost.  One 
way  estimates  can  be  inaccurate  is  if  they  fail  to  accurately  capture  inflation.  This  is 
especially  true  in  programs  of  longer  duration  or  high  dollar  value.  An  estimate  of 
inflation  prepared  by  the  DoD  Comptroller  is  included  in  all  cost  estimates.  If  inflation  is 
unexpectedly  high  in  a  given  year  then  that  forecasting  error  could  contribute  to  cost 
overruns. 

Acquisition  Reform 

In  an  attempt  to  make  the  DoD  more  responsive  and  efficient  in  the  procurement 
of  weapon  systems,  Congress,  the  DoD,  and  the  individual  military  services  have 
instituted  various  reforms  to  the  acquisitions  process.  Major  reforms  and  initiatives  from 
1971  to  2003  are  listed  in  Table  7.  However,  a  quick  glance  at  the  table  reveals  the 
challenge  in  trying  to  discern  the  impact  of  a  given  acquisition  refonns.  Almost  every 
year,  some  type  of  reform  effort  is  undertaken  and  it  becomes  difficult  to  identify  the 
major  reforms. 
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Table  7.  Acquisition  Initiatives  and  Legislation  (Scofield,  2003:19) 

Acquisition  Initiatives  and  Legislation 

1971  DODD  5000.1  (major  systems  acquisitions) 

1972  Commission  on  Government  Procurement 

1973  DODD  5000.4  (Cost  Analysis  Improvement 

Group) 

DODD  5000.3  (Test  &  Evaluation) 

1975  DODI  5000.2  (major  systems  acquisitions) 

DODD  5000.28  (design  to  cost) 

1976  OMB  Circular  A-1 09 

1978  Defense  Science  Board  Acquisition  Cycle  Task 

Force 

1979  Defense  Resource  Management  Study 

1981  Carlucci  Initiatives:  Defense  Acquisition 

Improvement  Program 

1982  Nunn-McCurdy  (thresholds) 

1983  Office  of  Federal  Procurement  Policy  Act 
Grace  Commission 

1984  DOD  Authorization  Act  (Public  Law  98-94) 

created  Office  of  Operational  Test  &  Evaluation 

1984  Competition  in  Contracting  Act  (CICA) 

1985  DOD  Procurement  Reform  Act 
DOD  5000.43  (streamlining) 

1986  Packard  Commission 
Goldwater-Nichols  Department  of  Defense 

Reorganization  Act 

Defense  Procurement  Improvement  Act 
Defense  Acquisition  Improvement  Act 

1987  DODD  5134.1  (Undersecretary  of  Defense. 

Acquisition) 

DODD  5000.49  (Defense  Acquisition  Board) 

1989  Defense  Management  Review 
Ethics  Reform  Act 

1990  Defense  Acquisition  Workforce  Improvement 

Act 

1991  Revised  DODI  5000.2  (major  systems  acquisitions) 

Section  800  Panel  created  by  1991  National 
Defense  Authorization  Act  (Public  Law  101-510) 

1994  Defense  Acquisition  Reform 

Federal  Acquisition  Streamlining  Act  (FASA) 

1995  Federal  Acquisition  Improvement  Act  (FASA  II) 

Air  Force  Lightning  Bolts  1 

1996  Federal  Acquisition  Reform  Act 
Clinger-Cohen  Act 

Rewrite  DOD  5000  Series 

1999  Air  Force  Lightning  Bolts  2 

2000  Revised  DOD  5000  Series 

2002  Agile  Acquisition  Initiatives  (Air  Force  Light¬ 

ning  Bolts  3) 

2003  Rewrite  DOD  5000  Series 


However,  past  research  illustrates  that  four  reforms  are  singular  in  their  scope  and 
potential  impact  on  cost  overruns.  Table  8  presents  a  list  of  those  key  acquisition 
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reforms,  laws,  and  studies  that  have  been  highlighted  by  past  research  and  based  on  a 
review  of  this  research,  four  are  selected  for  study. 


Table  8.  Key  Reforms  Identified  by  Past  Research 


Nunn-McCurdy  Act 

1982 

X 

1 

Grace  Commision 

1983 

X 

X 

2 

DoD  Authorization  Act 

1984 

X 

1 

Packard  Commission 

1986  X 

X 

X 

X 

X 

5 

Goldwater  Nichols  Act 

1986 

X 

1 

Defense  Management  Review 

1989 

X 

1 

DAWIA 

1990 

X 

1 

National  Performance  Review  1 

1993 

X 

1 

FASA 

1994 

X 

X 

2 

Cost  as  an  Independent  Variable 

1995 

X 

1 

Clinger-Cohen  Act 

1996 

X 

1 

Evolutionary  Acquisition 

2000 

X 

1 

Nunn  McCurdy  Act  of  1982 

The  Nunn  McCurdy  Act  of  1982  was  enacted  by  Congress  in  an  attempt  to 
control  the  spending  of  the  Department  of  Defense  and  to  force  the  DoD  to  provide 
information  to  Congress  when  a  major  program  experiences  significant  cost  overruns.  To 
accomplish  this  goal,  the  bill  called  for  “the  termination  of  weapons  programs  whose 
total  costs  grew  by  more  than  25  percent  above  original  estimates,  unless  they  were 
certified  as  critical  systems  by  the  Secretary  of  Defense  or  if  the  cost  growth  was 
attributable  to  certain  specified  changes  in  the  program”  (Center  for  Defense 
Information : 2 00 5 ) . 
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Such  a  clear  attempt  by  Congress  to  curtail  cost  overruns  in  defense  weapons 
systems  deserves  study.  Yet,  over  twenty  years  later  there  has  been  little  research  to 
evaluate  its  impact. 

Packard  Commission  of  1986 

According  to  Cooper  “the  ‘modern’  era  of  acquisition  refonn  commenced 
in  1986  with  the  President’s  Blue  Ribbon  Commission  on  Defense  Management  (Packard 
Commission)  (Cooper,  2002: 1 1).”  The  commission  was  chartered  by  President  Reagan 
in  late  1985  to  study  the  defense  acquisition  process  and  was  chaired  by  former  Deputy 
Secretary  of  Defense  David  Packard.  The  primary  conclusion  of  the  study  was  that 
“major  weapons  systems  cost  too  much,  take  too  long  to  field,”  and  as  a  result,  end  up 
fielding  “obsolete  technology”  (Searle,  1997:  32).  In  response  to  this,  the  Packard 
Commission  offered  the  following  recommendations  presented  in  Table  9: 

Table  9.  Packard  Commission  Recommendations 
(Modified  from  Searle,  1997:34) 

_ Packard  Commission  Recommendations _ 

Streamline  the  Acquisition  Process 
Increase  Tests  and  Prototyping 
Change  the  Organizational  Culture 
Improve  Planning 

Model  the  DoD  after  a  Competitive  Firm 


Unlike  the  Nunn-McCurdy,  act,  the  impact  of  the  Packard  Commission 
recommendation  have  been  extensively  researched  by  authors  such  as  Searle  (1997), 
Christensen  et  al.  (1999),  and  Holbrook  (2003).  And,  the  conclusion  drawn  by  all  of 
these  studies  is  virtually  the  same.  The  conventional  wisdom  is  that  either  the  Packard 
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Commission  recommendations  made  no  difference  (Holbrook:  2003:81)  or  that  the  cost 
performance  of  defense  contracts  actually  got  worse  in  the  period  following  Packard 
(Christensen  et  al,  1999:257). 

Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  of  1990 

“The  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  of 
1990  (Public  Law  101-510)  required  the  Secretary  of  Defense...  to  establish  education 
and  training  standards,  requirements,  and  courses  for  the  DoD  civilian  and  military 
workforce”  (Cooper,  2002: 12).  While  not  traditionally  viewed  as  an  acquisition  refonn, 
DAWIA  “shaped  the  way  education  and  training  is  provided  to  DoD  acquisition 
personnel. . .”  (Cooper,  2002: 12)  and  therefore  is  considered  by  this  research  to  be  worthy 
of  study.  Intuitively,  a  well-trained  acquisition  workforce  should  be  able  to  deliver 
increased  combat  capability  more  quickly  and  at  a  reduced  cost. 

Federal  Acquisition  Streamlining  Act  (FAS A)  of  1994 

The  Federal  Acquisition  Streamlining  Act  was  enacted  in  1994  and  was  “the  first 
major  rewrite  of  government  procurement  regulations  in  a  decade”  (Cooper,  2002:15). 
The  primary  intent  of  FAS  A  was  to  empower  the  acquisition  workforce  to  make 
decisions  about  how  to  manage  their  programs.  This  was  a  movement  away  from  the 
restrictive  and  centralized  procurement  practices  of  the  past  (Cooper,  2002: 15).  Other 
key  provision  of  FAS  A  were  the  increased  use  of  performance  based  contract  payments, 
and  the  push  towards  more  “commercial  or  off  the  shelf  products”  (Cooper,  2002: 18). 
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Most  important  for  this  study  were  the  requirements  that  contractors  have  a  certified  cost 
accounting  system  that  complies  with  accepted  cost  accounting  standards  and  also  the 
increased  scrutiny  of  a  contractor’s  past  performance  in  the  source  selection  criterion  of 
major  weapon  system  purchases.  Clearly  these  provisions  should  have  led  to  decreases 
in  weapon  system  cost  and  should  be  investigated.  Indeed,  in  their  1999  paper, 
Christensen  et  al.  suggest  that  the  impact  of  the  FASA  reform  is  worthy  of  study  but  that 
some  time  would  have  to  pass  before  its  impact  could  be  felt.  (Christensen  et  al, 
1999:258).  Now,  twelve  years  after  the  reform,  the  impact  can  be  modeled  and  studied. 

Clinger-Cohen  Act  of  1996 

The  Clinger-Cohen  Act  of  1996  is  an  extension  of  the  ideas  of  FASA.  So  similar 
are  the  provisions  of  Clinger-Cohen  that  Holbrook  used  the  phrase  FASA  II  to  describe  it 
(Holbrook,  2003:20).  As  such,  the  Clinger-Cohen  Act  and  FASA  are  modeled  as  a  single 
reform  in  this  study. 

Challenges  in  Implementing  Refonns 

One  challenge  in  evaluating  the  impact  of  acquisition  refonn  is  deciding  when  to 
consider  a  reform  to  be  implemented.  This  is  an  important  decision  if  the  effects  of  the 
reforms  are  to  be  studied  properly.  Clearly,  if  Congress  passes  a  law  restructuring  the 
defense  acquisition  process  today,  it  should  not  be  considered  fully  implemented 
tomorrow.  Intuitively,  one  would  expect  any  reform  of  a  large  bureaucracy  to  take  some 
time  to  be  implemented  and  an  even  longer  time  before  its  results  appeared  in  contract 
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cost  data.  Indeed,  organizational  behavior  literature  suggests  that  changes  can  take  years 
to  become  fully  implemented  within  a  large  bureaucracy  (Geert  et  al.,  2002: 11).  The 
issue  is  eloquently  described  by  Geert  et  al.  in  their  2002  working  paper  A  Framework  for 
Assessing  Commitment  to  Change.  Process  and  Context  Variables  of  Organizational 
Change. 

“Time  plays  at  least  in  two  ways  a  central  role  in  the  change  process.  First, 
implementation  of  change  goes  through  different  phases.  ..These  phases  take  time. 
Common  to  all  the  implementation  models  is  the  message  that  efforts  to  bypass 
these  phases  seldom  yield  a  satisfactory  result  (Armenakis  &  Bedeian,  1999: 
303).” 

In  an  organization  as  large  and  complex  as  the  Department  of  Defense,  one  would 
expect  acquisition  reforms  would  take  some  time  to  fully  implement.  Compounding  the 
problem  is  the  frequency  with  which  such  reforms  occur.  No  doubt,  different  reforms 
have  been  implemented  to  varying  degrees  and  with  varying  effectiveness.  In  his  2002 
Naval  Postgraduate  School  (NPS)  Thesis,  Michael  Cooper  alludes  to  this  problem  when 
he  lists  “Reduce  the  number  of  Reform  Initiatives”  (Cooper,  1997:97)  as  one  of  its 
recommendations.  The  paper  goes  on  to  emphasize  that  reforms  need  to  be  clearly 
identified  and  prioritized  to  the  workforce  (Cooper,  1997:97). 

In  his  2000  article  in  the  Acquisition  Review  Quarterly,  Raymond  W.  Reig 
confronts  the  problem  of  identifying  when  a  reform  has  been  implemented  and  attempts 
to  measure  or  baseline  the  effective  date  of  modern  acquisition  refonn.  Choosing  one 
date  as  the  effective  date  for  some  many  disparate  initiatives  proves  to  be  a  formidable 
challenge.  However,  he  does  settle  on  a  date  of  January  1996  as  the  date  that  acquisition 
reforms  “first  became  effective  within  the  field. . .”  (Reig,  2000:38).  He  then  states  that 
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any  program  with  a  Milestone  III  (Milestone  C)  that  occurred  after  July  1996,  would  be 
beneficially  affected  by  acquisition  reform.  However,  one  must  ask  how  reasonable  it  is 
to  choose  one  date  by  which  all  acquisition  reforms  are  considered  implemented. 


Acquisition  Reform  Cost  Research 

“Despite  the  implementation  of  more  than  two  dozen  regulatory  and 
administrative  initiatives,  there  has  been  no  substantial  improvement  in  the  cost 
performance  of  defense  programs  for  more  than  30  years.” 

(Christensen  etal.,  1999:252) 


The  second  major  area  of  research  on  cost  overruns  looked  at  whether  acquisition 
reforms  had  any  measurable  impact  on  cost  overruns.  The  consensus  among  this  research 
is  that  acquisition  reforms  had  no  measurable  impact  on  cost  overruns.  However,  this 
research  is  subject  to  some  limitations  including  a  subjective  treatment  date  and  omitted 
variable  bias.  Table  10  is  a  summary  of  key  research  in  this  area: 


Table  10.  Acquisition  Reform  Impact  on  Cost  Overruns 


Author  (Year) 

Main  Findings 

Searle  (1997) 

Cost  Overruns  worsened  after  the  Packard  Commission 
Cost  Estimating  Error  is  a  causal  factor  in  cost  overruns 

Christensen  et  al.  (1999) 

Holbrook  (2003) 

Acquisition  Reform  had  no  significant  Impact  on  contract 
cost  performance 

Phillips  (2004) 

Choosing  one  date  as  the  effective  date  of  acquisition  reform  is  a  convenient  tool 
for  analysis  employed  by  all  of  this  research.  Phillips  used  December  31,  1996  as  the 
treatment  date  for  comparing  pre-reform  and  post-reform  cost  growth.  Using  Selected 
Acquisition  Report  (SAR)  data  derived  from  the  1993  RAND  cost  growth  database  he 
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concluded  that  acquisition  reform  has  had  no  impact  on  cost  growth  (Phillips,  2004:70). 
Holbrook  arrived  at  a  similar  conclusion  looking  at  data  from  the  DAES  database.  Also 
looking  at  data  from  the  DAES  database,  Christensen,  Searle,  and  Vickery  used 
December  31,  1991  as  the  treatment  date  for  determining  the  impact  of  the  Packard 
Commission’s  recommendations  (Christensen  et  ah,  1999:254)  and  concluded  the 
following: 

“Based  on  a  review  of  269  completed  defense  contracts,  we  found  that  the 
Packard  Commission’s  recommendations  did  not  reduce  cost  overruns.  This 
result  is  consistent  with  similar  research  involving  an  analysis  of  cost  growth  on 
197  defense  programs  (Drezner,  Jarvaise,  Hough,  and  Norton,  1993).” 

Despite  the  apparent  clarity  of  these  results  there  is  some  question  in  the  wisdom  of 
choosing  a  single  date  to  represent  the  effective  date  of  all  acquisition  reforms  as  was  done 
by  Holbrook  and  Phillips.  Additionally,  none  of  these  studies  include  other  previously 
identified  variables  such  as  changes  in  defense  budgets,  industry  concentration,  inflation,  or 
war.  The  absence  of  these  variables  could  bias  the  result  of  their  research.  Christensen  et  al. 
recognized  this  issue  when  they  identified  threats  to  internal  validity  in  the  footnotes  of  their 
research  article. 

To  their  credit,  this  research  was  a  needed  first  step  in  studying  the  effectiveness 
of  acquisition  reform  and  the  researchers  did  recognize  that  additional  research  needed  to 
be  done.  For  example,  in  his  concluding  chapter,  Holbrook  states  “There  appears  to  be  a 
relationship  between  acquisition  reform  events  and  an  immediate  change  in  cost 
performance.  However. .  .this  study  cannot  provide  answers  as  to  why  these  changes 
occurred  and  why  they  appear  to  be  short  lived”  (Holbrook,  2004:  80).  Similarly,  Searle, 
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in  his  1997  thesis  suggested  that  future  research  “investigate  other  possible  causes  of  the 
dramatic  change  in  cost  performance  after  Dec  3 1,  1991.”  He  went  on  to  say  that 
“Perhaps  other  significant  factors  not  accounted  for  in  this  thesis  may  have  been 
responsible  for  the  changes  noted”  (Searle,  1997:89). 

Chapter  Summary 

This  chapter  identified  the  problem  of  cost  overruns  in  the  purchase  of  defense 
weapon  systems  and  reviewed  the  attempts  of  past  researchers  to  explain  its  causes.  As 
discussed  in  this  chapter,  past  research  identified  funding  instability,  war,  cost  estimating 
error,  and  defense  industry  consolidation  as  contributing  factors  but  has  not  been  able  to 
create  an  empirical  model  to  measure  their  relative  impact.  Also,  this  chapter  reviewed 
the  major  acquisition  reforms  of  the  last  20  years  and  discussed  the  challenge  in 
identifying  major  reforms  as  well  as  measuring  their  effective  date.  This  research  further 
looked  at  studies  that  evaluated  the  effectiveness  of  acquisition  reform  and  concluded  that 
while  the  consensus  among  previous  research  is  that  acquisition  reform  had  no  impact  on 
reducing  cost  overruns,  that  research  is  subject  to  some  limitations  which  potentially  bias 
their  results  and  suggest  that  further  study  is  needed. 
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ITT.  Data  and  Methods 


Chapter  Overview 

This  chapter  explains  the  methods  employed  in  this  research  effort  and  describes 
the  data  used  to  answer  the  research  questions  discussed  in  Chapter  I.  The  primary 
purpose  of  this  research  is  to  develop  an  empirical  model  that  can  be  used  to  study  the 
impact  of  acquisition  reforms,  changing  levels  of  defense  industry  consolidation  and 
defense  budgets,  unexpected  inflation,  and  war  on  the  cost  of  defense  weapon  systems. 
To  accomplish  this  goal,  data  was  collected  on  the  dependent  variable  (cost  overruns  of 
weapon  systems  contracts  for  Air  Force,  Army,  and  Navy)  and  the  independent  variables 
(defense  budgets,  industry  concentration  levels,  acquisition  reforms,  predicted  and  actual 
inflation  rates,  and  war). 

The  relationships  between  the  variables  are  then  described  using  fixed-effects 
panel  regression  models  to  describe  cost  overruns  in  fixed-price  and  cost-plus  contract 
types  as  well  as  the  production  and  research  and  development  program  phases.  This 
chapter  provides  a  brief  discussion  of  the  advantages  of  the  fixed-effect  panel  model  in 
cross-sectional  time  series  analysis,  an  overview  of  the  model  data,  and  a  detailed 
description  of  the  econometric  analysis  that  is  employed  in  the  next  chapter. 

Description  of  the  Models 

In  total,  four  models  are  presented.  Models  1  and  2  describe  how  cost  overruns 
relate  to  each  contract  type  and  Models  3  and  4  describe  cost  overruns  by  program  phase. 
Recall  that  past  researchers  such  as  Searle  (1997),  Christensen  et  al.  (1999),  and 
Holbrook  (2003)  also  studied  contract  cost  overruns  by  contract  type  and  program  phase. 
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This  is  because  the  different  types  and  phases  are  expected  to  behave  differently.  For 
example,  the  cost-plus  contract  type  is  expected  to  put  more  risk  on  the  government. 
Consequently,  the  overall  magnitude  of  cost  overruns  for  this  contract  type  is  expected  to 
be  greater. 

Advantages  of  the  Fixed-Effects  Panel  Model 

As  stated  above,  this  research  employs  a  fixed  effects  panel  model  to  analyze 
annual  cost  overruns  of  the  Air  Force,  Army,  and  Navy.  Since  the  cost  overruns  are  over 
time  and  involve  multiple  cross  sections,  a  panel  model  is  the  ideal  analytical  tool.  Cost 
overruns  could  have  been  studied  using  a  simple  time  series  regression  model  using 
ordinary  least  squares  regression.  However,  modeling  the  data  as  a  panel  model  “creates 
more  variability,  through  combining  variation  across  micro-units  with  variation  over 
time.”  This  increased  variability  makes  the  fixed-effects  panel  model  more  robust  to 
multicollinearity  (Kennedy,  2003:302). 

Perhaps  more  important  is  that  since  a  panel  model  accounts  for  heterogeneity  in 
the  cross-sectional  values,  it  can  correct  for  omitted  variables.  This  is  important  because 
omitted  variables  bias  the  regression  results.  (Kennedy,  2003:303)  Put  simply,  the  issue 
of  omitted  variables  is  a  serious  limitation  of  past  research  on  cost  overruns.  The  fact  that 
the  panel  model  can  account  for  this  problem  is  perhaps  the  biggest  single  advantage  of 
this  technique. 
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Data 

Dependent  Variable 

The  dependent  variable  in  this  analysis  is  the  annual  percentage  of  contract  cost 
overruns  on  ACAT  I  weapons  programs  for  the  United  States  Air  Force,  Army  and  Navy 
from  1970  to  2002.  Contract  cost  overruns  were  measured  using  Earned  Value 
Management  (EVM)  data  from  the  Defense  Acquisition  Executive  Summary  (DAES) 
database.  Cost  overruns  from  contracts  contained  in  the  DAES  database  were  separated 
by  service  and  by  contract  type  and  phase  and  then  used  to  calculate  average  percentage 
contract  cost  overruns  per  year. 

There  are  two  main  sources  of  cost  information  for  defense  weapon  systems:  The 
Selected  Acquisition  Reports  (SAR)  and  the  Defense  Acquisition  Executive  Summary 
Database  (DAES).  SARs  were  created  by  the  Department  of  Defense  in  1967  in  order  to 
generate  a  consistent  database  of  cost,  schedule,  and  performance  infonnation  on  major 
defense  acquisition  programs.  The  DAES  database  is  a  collection  of  cost  information  on 
ACAT  1  programs  by  acquisition  contract  (Holbrook,  2003:33).” 

Compilations  of  SAR  cost  and  schedule  information  have  been  used  extensively 
in  past  cost  research.  However,  due  to  the  highly  aggregated  nature  of  the  data  in  the 
SAR  database  (Christensen  et  al,  1999:252),  there  are  several  limitations  to  this  database 
that  reduce  its  effectiveness  in  explaining  cost  growth  (Jarvaise  et  al.,  1996:11). 
Consequently,  for  this  research  the  DAES  database  is  used. 

In  the  DAES  database  is  the  Earned  Value  Management  (EVM)  data  derived  from 
contractor  reports  called  Cost  Performance  Reports  (CPRs).  EVM  is  a  tool  used  by 
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contractors  and  the  government  to  track  costs  on  defense  contracts  (Holbrook,  2003:25). 
Since  each  EVM  system  is  validated  by  the  government  prior  to  use,  the  data  from  the 
EVM  system  is  considered  valid  (Searle,  1997:42). 

The  cost  data  used  in  this  study  covers  ACAT  I  contracts  from  1970  to  2002  and 
consists  of  14,003  entries  from  186  ACAT  I  programs  from  the  Air  Force,  Army  and 
Navy.  Table  11  shows  the  number  of  programs  per  service  and  Figure  3  is  an  excerpt 
from  the  database: 


Table  11.  Number  of  Programs  per  Service 


Service 

Number  of  Programs 

Air  Force 

61 

Army 

50 

Navy 

75 

SUBMITDATE  | Contract  ID 

Service 

ProgramName  |ACWP 

BCWP 

BCWS 

BAC  |  Contract  Type  |  Program  Phase 

3/25/1976 

N0003074C0100 

Navy 

TRIDENT  SUB 

1005.5 

973.7 

987.9 

2028.6 

CP 

DEVELOPMENT 

4/25/1976 

N0001975C0424 

Navy 

F/A-18  C/D 

20.6 

21.8 

21.9 

1020.2 

CP 

DEVELOPMENT 

5/25/1976 

DAAK4072C01 06 

Army 

PATRIOT 

532.7 

522 

529.7 

569.2 

CP 

DEVELOPMENT 

5/25/1976 

DAAK4072C0773 

Army 

STINGER 

64.5 

53.6 

55.3 

62.9 

CP 

DEVELOPMENT 

6/25/1976 

F1962870C0218 

Air  Force 

E-3A  Hawkeye 

697 

662.2 

673.9 

768.3 

CP 

DEVELOPMENT 

6/25/1976 

F1962870C0218 

Air  Force 

E-3A  Hawkeye 

142.9 

142.2 

146.6 

259.3 

FPI 

PRODUCTION 

6/25/1976 

F1962874C0127 

Air  Force 

E-4  (AABNCP) 

80.4 

81.7 

83.4 

181.3 

CP 

DEVELOPMENT 

6/25/1976 

N0001976C0261 

Navy 

F/A-18  C/D 

23.3 

22.9 

23.1 

330.6 

CP 

DEVELOPMENT 

6/25/1976 

F3365770C0300 

Air  Force 

F-15 

827 

803 

804 

809.3 

CP 

DEVELOPMENT 

6/25/1976 

F3365775C0310 

Air  Force 

F-16 

114.2 

109.1 

127.7 

389.5 

FPI 

DEVELOPMENT 

6/25/1976 

N0003074C0100 

Navy 

TRIDENT  SUB 

1105.4 

1084.2 

1087.5 

2026.2 

CP 

DEVELOPMENT 

7/25/1976 

F3365775C0310 

Air  Force 

F-16 

127.1 

119.8 

140 

399 

FPI 

DEVELOPMENT 

8/25/1976 

DAAJ01 75C0360 

Army 

AH-64  Apache 

12.8 

13 

11.6 

32.5 

CP 

DEVELOPMENT 

8/25/1976 

N0001975C0267 

Navy 

C/MH-53E 

36.6 

32.7 

34.5 

69.7 

CP 

DEVELOPMENT 

8/25/1976 

DAAA0976C2001 

Army 

COPPERHEAD 

11.6 

11.3 

11.5 

41.6 

CP 

DEVELOPMENT 

8/25/1976 

N0001975C0424 

Navy 

F/A-18  C/D 

49.5 

51.9 

52.9 

1022.2 

CP 

DEVELOPMENT 

8/25/1976 

F3365776C0100 

Air  Force 

F-15 

135.6 

136.9 

134.8 

852.9 

FPIF 

PRODUCTION 

8/25/1976 

F0470475C0014 

Air  Force 

Minuteman  III 

18.4 

19.4 

18.6 

75.8 

FPI 

PRODUCTION 

Figure  3.  Excerpt  from  DAES  Database 


A  complete  list  of  programs  studied  and  the  number  of  contract  entries  per  program  can 
be  found  in  Appendix  A. 
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To  identify  the  lead  service  and  program  name,  an  identifier  from  the  DAES 
database  called  a  PNO  number  was  matched  with  a  list  of  PNO  numbers  compiled  by 
Carden  in  the  completion  of  his  2006  AFIT  Master’s  Thesis.  Data  was  then  further 
grouped  by  contract  type  into  fixed-price  contracts  and  cost-plus  contracts.  Contracts 
that  were  a  mix  of  the  two  types  or  contracts  where  the  contract  was  unknown  were  not 
included  in  either  the  fixed-price  contract  model  or  the  cost-plus  contract  model. 
Additionally,  the  contracts  were  segregated  by  program  phase  into  production  contracts 
and  research  and  development  (R&D)  contracts.  Again,  contracts  where  the  phase  was 
unknown  were  not  included  in  either  of  these  models.  Contracts  where  no  lead  service 
was  known  or  contracts  that  were  managed  by  the  Office  of  the  Secretary  of  Defense 
(OSD)  were  also  not  included  in  this  study.  Table  12  shows  the  number  of  contract 
entries  in  the  database  per  contract  type: 


Table  12.  Number  of  Contract  Entries  per  Service  and  Contract  Type 


Service 

Fixed  Price 

Cost-Plus 

Grand  Total 

Air  Force 

3485 

1588 

5073 

Army 

907 

1839 

2746 

Navy 

3243 

2247 

5490 

Grand  Total 

7635 

5674 

13309 

Table  13  shows  the  number  of  contract  entries  in  the  database  per  program  phase: 


Table  13.  Number  of  Contract  Entries  per  Service  and  Program  Phase 


Service 

Production 

Development 

Grand  Total 

Air  Force 

2267 

2151 

4418 

Army 

855 

1385 

2240 

Navy 

2924 

1418 

4342 

Grand  Total 

6046 

4954 

11000 
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The  dependent  variable  in  this  analysis  is  the  annual  percentage  of  contract  cost 
overruns  per  service  per  year.  These  values  were  derived  from  the  EVM  data  of  each 
contract  entry  by  first  using  the  following  formula  for  percentage  contract  cost  variance 


(%CV)  from  the  Earned  Value  Management  Gold  Card  on  all  contract  entries: 


%CV  =  (ACWP-BCWP) 
BCWP 


(1) 


where:  %CV  =  Percentage  Cost  Variance 

BCWP  =  Budgeted  Cost  of  Work  Performed 
ACWP  =  Actual  Cost  of  Work  Performed 

Then,  the  %CV  from  each  contract  entry  is  aggregated  by  year,  service,  program  type, 
and  program  phase  and  an  annual  cost  overrun  percentage  is  calculated  using  a  weighted 
average  of  the  individual  %CV’s.  Graphs  of  cost  overruns  over  time  and  for  each  model 
are  found  in  Chapter  IV. 

Independent  Variables 

Table  14  lists  the  independent  variables  used  in  this  analysis.  A  discussion  of 
each  variable  type  follows. 
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Table  14.  List  of  Independent  Variables 


Independent  Variables 

Variable  Description 

Variable  Name 

Percent  Change  in  Operations  and  Maintenance  Budget 

Percent  Change  in  Procurement  Budget 

Percent  Change  in  Research  and  Development  Budget 

Industry  Concentration  (CR4) 

Unexpected  Inflation 

Packard  Commission 

Federal  Acquisition  Streamlining  Act  (FASA) 

Nunn-McCurdy  Act 

Defense  Acquisition  Workforce  Improvement  Act  (DAWIA) 

War 

ombudgetpercent 

procbudgetpercent 

rdbudgetpercent 

concentrationcr4 

inflationdifference 

packard 

fasa 

nunnmccurdy 

dawia 

war 

Defense  Budgets 

In  the  models  presented  in  this  research,  the  annual  percentage  change  in  defense 
budgets  is  a  proxy  for  funding  instability.  Annual  budget  data  is  obtained  from  Chapter  6 
of  the  National  Defense  Budget  Estimates  for  2006,  a  report  prepared  annually  by  the 
Office  of  the  Undersecretary  of  Defense  (Comptroller).  Budget  data  from  this  document 
is  collected  per  cross-sectional  unit  (Air  Force,  Army,  and  Navy),  and  per  appropriation. 
For  the  purposes  of  this  study,  the  budget  remaining  after  Procurement  and  R&D  funds 
are  subtracted  is  referred  to  as  the  Operations  and  Maintenance  (O&M)  budget.  Finally, 
budget  values  are  then  converted  into  annual  percentage  changes  to  be  consistent  with  the 
dependent  variable  and  for  ease  of  interpretation  of  model  coefficients.  A  table  of 
budget  amounts  for  each  service  is  presented  in  Appendix  C. 


Acquisition  Reforms 

The  acquisition  refonns  studied  in  this  research  are  coded  as  dummy  variables 
with  a  “1”  representing  the  presence  of  the  reform  and  a  “0”  indicating  that  the  refonn  is 
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not  in  effect.  The  Nunn-McCurdy  act  was  enacted  in  1982  so  it  is  coded  as  a  “1”  from 
1982  until  2002.  The  Packard  Commission  recommendations  were  issued  in  1986  so 
from  1986  until  1994,  Packard  is  coded  as  a  “1”.  In  1994,  the  Federal  Acquisition 
Streamlining  Act  (FASA)  was  enacted  and  superseded  many  of  the  Packard  Commission 
reforms  so  after  1994,  Packard  is  coded  as  a  “0”  and  from  1994  until  2002,  FASA  is 
coded  as  a  “1”.  Finally,  in  1990,  the  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA)  became  law  so  from  1990  until  2002,  DAWIA  is  coded  as  a  “1”. 

War 

War  is  also  modeled  as  a  dummy  variable  and  is  coded  as  a  “1”  for  the  Vietnam 
War,  the  Gulf  War,  the  conflict  in  Bosnia  in  1999,  and  the  period  after  September  1 1th. 

Industry  Consolidation 

The  level  of  consolidation  in  the  defense  industry  is  measured  by  an  industry 
concentration  ratio  called  the  CR4.  The  CR4  measures  “how  much  of  the  total  output  of 
an  industry  is  produced  by  the  largest  (four)  firms  in  that  industry. . . .  When  an  industry 
is  composed  of  a  very  large  number  of  firms,  each  of  which  is  very  small,  the  (CR4)  is 
close  to  zero.  When  four  or  fewer  firms  produce  all  of  an  industry’s  output,  the  (CR4)  is 
close  to  1”  (Baye,  2003:233). 

CR4  data  was  gathered  from  the  Economic  Census  reports  generated  every  five 
years  by  the  US  Census  Bureau.  For  the  purposes  of  this  study,  North  American 
Industry  Classification  System  (NAIC)  category  366411,  Aerospace  Products  and  Parts 
Manufacturing  represents  the  industry  concentration  of  the  Air  Force,  the  Army 


29 


concentration  level  is  represented  by  NAIC  3795,  Tank  and  Tank  Components,  and  the 
Navy  is  represented  by  NAIC  3731,  Ship  Building  and  Repair.  Figure  4  is  a  graph  of  the 
concentration  ratios  from  1970-2002: 

DoD  Industry  Concentration  (CR4) 


-♦-CR4  (Air  Force)  -»-CR4  (Army)  CR4  (Navy) 


Figure  4.  Industry  Concentration  Ratios  by  Service 


Unexpected  Inflation 

Recall  that  cost  estimating  error  was  identified  in  past  research  as  a  causal  factor 
in  cost  overruns  (Christensen  et  ah,  1999:25 1).  One  factor  that  contributes  to  cost 
estimating  error  is  error  in  the  inflation  prediction  of  a  cost  estimate.  Future  inflation  of 
defense  spending  is  forecasted  by  OSD  Comptroller  and  as  they  discuss  in  Chapter  5  of 
National  Defense  Budget  Estimates  for  2006,  forecasts  of  inflation  are  adjusted  several 
times  before  the  final  forecast.  Even  then,  the  forecast  does  not  exactly  match  the  actual 
inflation  in  that  year. 
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Given  this  discrepancy,  this  research  models  a  variable  called  Unexpected 


Inflation  and  tests  whether  it  is  correlated  with  cost  overruns.  Unexpected  inflation  is 
defined  as  the  difference  between  actual  inflation  and  expected  inflation  in  the  year  the 
money  is  used. 

In  reality,  each  service  and  appropriation  has  a  different  annual  prediction  of 
expected  inflation  and  a  different  actual  inflation.  However,  for  this  model,  the  aggregate 
difference  is  used  for  each  cross  section  (Air  Force,  Army,  and  Navy).  Table  18  displays 
the  expected  and  actual  inflation  values  for  each  year  as  well  as  the  calculated  difference 
referred  to  in  this  research  as  Unexpected  Inflation.  The  availability  of  historical  budget 
and  forecast  data  at  the  time  of  this  research  precludes  the  use  of  values  before  1980. 


Table  15.  Unexpected  Inflation  in  DoD 


Actual  Inflation 

Expected  Inflation 

Unexpected  Inflation 

1980 

11.7% 

5.9% 

5.8% 

1981 

10.4% 

8.1% 

2.3% 

1982 

7.5% 

8.9% 

-1.4% 

1983 

3.6% 

6.3% 

-2.7% 

1984 

3.0% 

3.7% 

-0.7% 

1985 

3.3% 

4.5% 

-1.2% 

1986 

2.6% 

4.0% 

-1.4% 

1987 

2.9% 

3.4% 

-0.5% 

1988 

3.6% 

4.5% 

-0.9% 

1989 

3.9% 

3.4% 

0.5% 

1990 

3.0% 

3.0% 

0.0% 

1991 

4.6% 

4.0% 

0.6% 

1992 

1.9% 

2.9% 

-1.0% 

1993 

2.9% 

3.7% 

-0.8% 

1994 

2.3% 

2.0% 

0.3% 

1995 

2.0% 

2.2% 

-0.2% 

1996 

2.2% 

2.8% 

-0.6% 

1997 

2.2% 

2.6% 

-0.4% 

1998 

2.3% 

2.2% 

0.1% 

1999 

2.2% 

2.0% 

0.2% 

2000 

2.5% 

2.2% 

0.3% 

2001 

3.0% 

2.8% 

0.2% 

2002 

2.7% 

3.0% 

-0.3% 
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Methods 


The  model  proposed  in  this  research  is  that  contract  cost  overruns  are  a  function 
of  budgets,  industry  concentration,  inflation,  acquisition  reform,  and  war  such  that: 


COST  OVERRUN  %  = 


/ 


'FE-  +  AO  &  M  Budget  +  AProcurement  Budget  +  AR  &  D  Budget 
+  Industry  Concentration  +  Unexpected  Inflation  +  Packard 


|^+  FASA  +  NunnMcCurdy  +  DAWIA  +  War  +  AR1  disturbance  +  error  J 


(2) 


The  fixed-effects  panel  model  notation  is: 

yt,=a  +  xttp  +  vi+eit 


(3) 


where 


i  =  {Air  Force,  Army,  Navy} 
j  =  {1970, 1971,. ..2002} 


The  dependant  variable  is  assumed  to  have  lag  1  autocorrelation  so  an  AR  1 
disturbance  term  is  added  to  the  model  (Stata,  2005:3 1 1).  Consequently,  the  error  tenn  is 
represented  as  follows: 


£u=  P£u-i+rl  u 


(4) 


Using  this  notation,  the  basic  representation  of  all  four  models  is: 


COST  OVERRUN  %  =  a;  +  P,AO  &  M  Budgetit  +  (RAProcurement  Budgetit  + 
(33AR  &  D  Budgetit  +  (34Industry  Concentration;, 

+  P5Unexpected  Inflation;,  +P6Packardit  +  P7FASA;, 

+  (3gNunnMcCurdyit  +  P9DAWIAit  +  P10Warit  +  V;  +  sit 
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The  first  step  in  this  analysis  was  to  check  for  stationarity  of  the  dependent 
variable  for  each  of  the  four  models.  According  to  Kennedy,  regression  analysis  of  panel 
data  could  result  in  spurious  correlations  if  the  dependent  variable  is  non-stationary 
(Kennedy,  2003:325).  This  research  tests  for  stationarity  by  two  methods.  The  first 
method  is  by  inspection  of  a  time  series  plot  of  each  dependent  variable.  The  second 
method  is  a  more  formal  test  called  the  Augmented  Dickey  Fuller  Test.  The  Augmented 

Dickey-Fuller  test  tests  if  a  variable  “follows  a  unit  root  process . The  null  hypothesis 

is  that  the  variable  contains  a  unit  root,  and  the  alternative  is  that  the  variable  was 
generated  by  a  stationary  process”  (Stata,  2005). 

According  to  Makridakis  et  al.  in  their  Forecasting  Methods  and  Applications 
text,  the  four  assumptions  of  multiple  linear  regression  are  model  form,  independence  of 
residuals  (lack  of  autocorrelation),  constant  variance  (homoskedasticity),  and  nonnality 
of  residuals  (Forecasting,  1998:259).  However,  a  reading  of  Kennedy’s  A  Guide  to 
Econometrics  illustrates  that  the  fixed-effects  panel  model  is  robust  to  normality  and  is 
also  effective  at  dealing  with  omitted  variable  bias  that  would  make  model  fonn  a  more 
serious  concern.  Therefore,  this  research  tests  each  panel  model  only  for  independence 
or  a  lack  of  autocorrelation  and  homoskedasticity  or  constant  variance. 

Autocorrelation  means  that  model  residuals  are  not  independent  of  their  lagged 
values  (Forecasting,  1998:265).  The  presence  of  autocorrelation  in  time  series  models 
such  as  the  panel  model  is  a  serious  concern  to  econometricians  (Kennedy,  2003:134) 
because  it  introduces  bias  to  the  model  coefficients.  To  measure  autocorrelation,  the 
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Baltagi  Wu  Locally  Best  Invariant  (LBI)  test  was  calculated.  The  Baltagi-Wu  LBI  is  an 
extension  of  the  Durbin- Watson  statistic  to  unbalanced  panel  models  and  tests  a  model 
for  lag  1  autocorrelation  (Stata,  Longitudinal/Panel  Data,  2005:316). 

Like  the  Durbin-Watson  statistic,  the  Baltagi-Wu  LBI  ranges  from  0  to  4  with 
values  significantly  lower  or  higher  than  2  indicating  autocorrelation.  The  further  the 
calculated  value  is  away  from  2,  the  more  evidence  of  positive  (less  than  2)  or  negative 
(greater  than  2)  autocorrelation  of  the  dependant  variable.  No  p-values  are  available  for 
the  Baltagi-Wu  LBI,  so  the  presence  of  autocorrelation  is  a  subjective  detennination  of 
the  researcher.  In  the  event  that  the  AR(1)  disturbance  term  does  not  adequately  account 
for  the  remaining  autocorrelation,  lagged  dependent  variables  are  added  until  the  Baltagi- 
Wu  LBI  approaches  2.0 

To  test  for  homoskedasticity,  the  Breusch-Pagan  test  is  employed.  The  null 
hypothesis  of  the  Breusch-Pagan  test  is  that  the  variance  is  equal  to  zero. 
Heteroskedasticity  “. .  .does  not  lead  to  biased  coefficients.  But  when  the 
homoskedasticity  assumption  is  violated,  the  coefficients  of  a  regression  model  are  not 
efficient...”  (Koeler  and  Kreuter,  2005:214). 

Finally,  lagged  values  of  the  acquisition  reform  and  budget  variables  were  added 
to  the  specified  model.  While  it  is  expected  that  budgets  and  acquisition  reforms  are 
correlated  with  cost  overruns,  past  research  suggests  that  the  effect  may  take  a  while  to 
materialize.  Intuitively,  this  makes  sense.  Changes  in  the  current  year’s  budget  are  not 
likely  to  have  as  much  of  an  impact  on  existing  contracts  as  would  changes  in  the 
previous  number  of  years.  Similarly,  with  acquisition  reform,  organizational  behavior 
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literature  as  well  as  past  cost  research  suggests  that  reforms  take  years  to  be 
implemented. 

As  stated  by  Kennedy  in  A  Guide  to  Econometrics,  “It  is  common  for 
practitioners  to  use  selection  criteria  such  as  the  Akaike  Information  Criterion  (AIC)  or 
adjusted  R  to  aid  in  model  specification  particularly  in  detennining  things  like  the 
number  of  lags  to  include  (Kennedy,  2003:88).”  Based  on  this  guidance,  this  research 
used  the  AIC  to  determine  the  lag  structure  for  the  budget  and  acquisition  reform 
variables. 

Summary 

This  research  analyzes  the  cost  overruns  of  defense  weapon  system  contracts  by 
contract  type  and  program  phase  over  time.  This  chapter  describes  how  a  fixed-effects 
panel  model  is  used  to  create  these  empirical  models.  Additionally,  this  chapter 
described  how  the  data  for  the  dependent  and  independent  variables  was  gathered  and 
prepared.  Finally,  this  chapter  outlined  the  process  of  analysis  that  is  undertaken  on  each 
of  the  four  models  in  the  next  chapter. 
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IV.  Analysis  and  Results 


Chapter  Overview 

Recall  from  previous  chapters  that  cost  overruns  in  defense  weapons  systems  are 
hypothesized  to  be  correlated  with  changes  in  defense  budgets,  levels  of  concentration  in 
the  defense  industry,  unexpected  inflation,  acquisition  reform,  and  war.  Table  16  is  a 
summary  of  the  independent  variables  and  their  expected  impact  on  cost  overruns  derived 
from  a  review  of  past  research. 


Table  16.  Summary  of  Independent  Variables 


Factor 

Model  Variable 

Expected  Impact 

Funding  Instability 

Percentage  Change  in  Defense  Budgets 

+ 

Industry  Consolidation 

Industry  Concentration  (CR4) 

- 

War 

War  Dummy  (Vietnam,  Gulf  War,  Bosnia,  Post- 
September  1 1th) 

+ 

Cost  Estimating  Error 

Unexpected  Inflation  in  DoD 

+ 

Acquisition  Reform 

Dummy  Variables  for  Nunn-McCurdy,  Packard, 
FASA,  DAWIA 

Inconclusive 

This  chapter  presents  the  analysis  described  in  Chapter  III  and  is  organized  by 
model  in  the  following  manner: 


Table  17.  Order  of  Model  Presentation 


Model 

Description 

Model  1 

Fixed  Price  Cost  Overruns 

Model  2 

Cost-Plus  Cost  Overruns 

Model  3 

Production  Cost  Overruns 

Model  4 

Research  and  Development  Cost  Overruns 

The  general  model  takes  the  form  of  an  unbalanced  panel  model  with  cost 
overruns  as  the  dependent  variable.  Additionally,  the  model  has  an  AR(1)  disturbance 
tenn  represented  as  Vi  to  account  for  autocorrelation: 
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COST  OVERRUN  %  =  a;  +  (3, AO  &  M  Budget;,  +  (32AProcurement  Budgetit  + 

[TAR  &  D  Budgetit  +  (34Industry  Concentration it  ^ 

+  [^Unexpected  Inflation;,  +  [36Packardn  +  (37FASAit 
+  (38NunnMcCurdyit  +  (39DAWIA;,  +  (310War;,  +  v,  +  8it 

As  described  in  Chapter  III,  the  dependent  variable  of  each  model  was  checked 
for  stationarity  and  in  all  four  models,  the  data  was  found  to  come  from  a  stationary 
process.  Additionally,  all  model  residuals  were  tested  for  heteroskedasticity  and  found  to 
have  a  constant  variance.  All  four  models  were  found  to  have  significant  autocorrelation 
so  an  AR(1)  disturbance  term  was  added  to  each  model.  This  was  sufficient  in  all  but  the 
research  and  development  model  (Model  4)  in  which  the  first  two  lagged  dependent 
variables  were  added.  With  this  specification  change,  the  R&D  model  had  a  Baltagi-Wu 
LBI  of  2.07  which  indicates  that  the  autocorrelation  was  properly  accounted  for. 

As  the  model  results  are  discussed  and  interpreted,  recall  the  expected  direction  of 
impact  of  each  variable  and  also  the  four  research  questions  presented  in  Chapter  I. 
Essentially,  the  research  questions  ask  two  things:  how  are  decreases  in  budgets  (which 
proxy  funding  instability),  industry  concentration,  unexpected  inflation,  and  war 
correlated  with  cost  overruns  and  did  acquisition  reforms  have  a  measurable  impact  on 
those  overruns  once  the  other  factors  are  considered. 

Model  1:  Contract  Cost  Overruns  (Fixed  Price  Contracts) 

The  dependent  variable  in  this  model  is  the  average  annual  cost  overrun  per 
service  per  year  for  fixed-price  contracts.  Figure  5  is  a  visual  depiction  of  these  overruns. 
Notice  the  significant  increases  and  decreases  in  cost  overruns  from  year  to  year.  This  is 
the  behavior  that  the  empirical  model  seeks  to  explain. 
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Cost  Overruns  (Fixed  Price  Contracts) 


•  -  Air  Force  - ■ -  Army 

♦  Navy 


Figure  5.  Cost  Overruns  of  Fixed-Price  Contracts  (1970-2002) 


As  suggested  in  Chapter  III,  acquisition  reforms  and  budgets  are  expected  to  be 
most  significant  as  lagged  independent  variables.  The  lag  structure  for  model  1  was 
determined  by  minimizing  the  Akaike  Information  Criterion  (AIC)  and  is  presented  in 
Table  18.  For  more  on  how  the  lags  were  determined  see  Appendix  D. 


Table  18.  Model  1  Lag  Structure 


Model  1  Lag  Structure  (Fixed  Price  Cost  Overruns) 

Variable 

Using  AIC 

Packard 

0 

FASA 

0 

DAWIA 

2 

Nunn-McCurdy 

0 

Budgets 

0 
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From  the  regression  results  in  Table  20  below,  the  Baltagi-Wu  LBI  test  for  lag  1 
autocorrelation  has  a  value  of  1.22.  This  appears  to  indicate  some  remaining 
autocorrelation  of  the  residuals.  However,  since  an  AR(1)  disturbance  term  is  already 
included  in  the  model,  no  additional  specification  changes  are  made. 

Using  the  Breusch-Pagan  test  shown  in  Table  19  below,  the  prob>chi2  is  greater 
than  a  =  .05  ,  therefore  we  fail  to  reject  the  null  hypothesis  of  constant  variance  and 
conclude  that  the  model  does  not  suffer  from  heteroskedasticity. 

Table  19.  Test  for  Constant  Variance  (Model  1) 

_ Model  1:  overrunpercentfplOO _ 

Breusch-Pagan  Test  (H0=Constant  Variance) 


Estimated  Results  Variance 

Standard  Deviation 

overrunpercent  100 

79.13309 

8.895678 

e 

35.06199 

5.921317 

u 

0 

0 

chi2(l)  = 

0.47 

prob>chi2 

0.4928 

Using  the  lag  structure  from  Table  18  above,  the  Fixed-Effects  Panel  model 
regression  results  are  as  follows: 
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Table  20.  Panel  Results  Model  1:  Fixed-Price  Cost  Overruns 


Model  1:  Panel  Results  (Fixed-Price  Cost  Overruns) 

iFixed-effects  (within)  Regression  with  AR(1)  Disturbance 

Number  of  obs  = 

Group  variable  (i):  service 

Number  of  groups  = 

R-sq: 

within  = 

0.4774 

Obs  per  group:  min  = 

22 

between  = 

0.807 

avg  = 

22 

overall  = 

0.5659 

max  = 

22 

F(10,53) 

4.84 

Prob  >  F  = 

0.0001 

overrunpercentfp  100 

Coef. 

Std.  Err. 

t 

P>|t| 

O&M  Budget  (%Change) 

-31.5013 

11.4969 

-2.7400 

0.0080 

Procurement  Budget  (%  Change) 

-42.8579 

12.3519 

-3.4700 

0.0010 

R&D  Budget  (%  Change) 

-52.4508 

14.5825 

-3.6000 

0.0010 

Industry  Concentration  (CR4) 

0.1516 

0.3572 

0.4200 

0.6730 

Unexpected  Inflation 

134.0035 

92.9629 

-1.4400 

0.1550 

Packard  Commision  (Lag  5) 

-6.3498 

3.5810 

-1.7700 

0.0820 

FASA  (Lag  2) 

-21.9962 

4.8244 

-4.5600 

0.0000 

Nunn-McCurdy  Act  (Lag  7) 

-9.1006 

3.9929 

-2.2800 

0.0270 

DAWIA 

13.0930 

3.3530 

3.9000 

0.0000 

War 

1.2876 

1.6230 

0.7900 

0.4310 

_cons 

11.7176 

11.0470 

1.0600 

0.2940 

Baltagi-Wu  LBI 

1.2163 

The  results  of  this  analysis  show  that  budgets  are  negatively  correlated  with  cost 
overruns  as  are  the  Packard  Commission  reforms,  the  Federal  Acquisition  Streamlining 
Act  (FASA),  and  the  Nunn-McCurdy  act  while  the  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA)  is  positively  correlated  with  cost  overruns.  Table  21 
compares  the  actual  impact  of  each  independent  variable  on  cost  overruns  of  Fixed-Price 
contracts  with  the  expected  impact  from  past  research.  Independent  variables  that  are  not 
significantly  correlated  with  cost  overruns  are  left  blank  in  the  actual  impact  column. 
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Table  21.  Comparison  of  Expected  and  Actual  Impact  on  Cost  Overruns 


Model  1  (Fixed-Price  Cost  Overruns) 

INDEPENDENT  VARIABLES 

Expected  Impact  from 
Past  Research 

Actual 

Impact 

%  A  Operations  and  Maintenance  Budget 

%AProcurement  Budget 

_ 

_ 

%  AResearch  and  Development  Budget 

- 

- 

Industry  Concentration  (CR4) 

- 

Unexpected  Inflation 

+ 

Packard  Commission 

+ 

- 

FASA 

- 

Nunn-McCurdy  Act 

_ 

DAWIA 

+ 

War 

+ 

Notice  that  the  percentage  change  in  budgets  had  the  hypothesized  impact  but  that 
the  other  variables  were  either  not  significant  or  had  an  impact  different  than  what  was 
expected  from  past  research.  Most  notable  is  that  the  Packard  commission  reforms  are 
correlated  with  a  decrease  in  cost  overruns.  This  is  contrary  to  other  finding  such  as 
those  by  Searle  (1997),  Christensen  et  al.  (1999)  and  Holbrook  (2003).  Those  studies 
either  concluded  that  the  Packard  Commission  recommendations  were  correlated  with  an 
increase  in  cost  overruns  or  they  were  unable  to  show  any  significant  relationship. 

The  difference  in  this  study  is  the  inclusion  of  other  relevant  variables  such  as 
changing  defense  budgets.  In  effect,  this  result  suggests  that  the  Packard  Commission 
and  many  other  acquisition  reforms  would  have  reduced  cost  overruns  had  the  other 
factors  such  as  decreasing  defense  budgets  not  overwhelmed  their  impact. 

Indeed,  a  quick  glance  at  the  summary  of  regression  results  in  Table  22  reveals 
that  the  budget  variable  which  is  a  proxy  for  funding  instability  has  a  much  greater 
impact  on  cost  overruns  than  the  acquisition  reforms 


41 


Table  22.  Fixed  Price  Cost  Overrun  Model  Results 


Model  1  (Fixed-Price  Cost  Overruns) 

within  R2 

=.4774 

INDEPENDENT  VARIABLES 

Coefficient 

p-value 

%AOperations  and  Maintenance  Budget 

-31.5013 

0.0080 

%AProcurement  Budget 

-42.85786 

0.0010 

%AResearch  and  Development  Budget 

-52.45078 

0.0010 

Industry  Concentration  (CR4) 

0.1515838 

0.6730 

Unexpected  Inflation 

134.0035 

0.1550 

Packard  Commission 

-6.349819 

0.0820 

FASA 

-21.99616 

0.0000 

Nunn-McCurdy  Act 

-9.100645 

0.0270 

DAWIA 

13.09299 

0 

War 

1.287618 

0.4310 

In  this  model,  if  budgets  decrease,  cost  overruns  of  fixed  price  contracts  increase. 

Since: 

COST  OVERRUN  %  =  a,  +  P,  AO  &  M  Budgetit  +  P  ,AProcurement  Budgetit  + 

P,AR  &  D  Budgetit  +  P4Industry  Concentration  „ 

+  P5UnexpectedInflationit  +P6Packard„  +P7FASAit 
+  PsNunnMcCurdyit  +P,DAWIAit  +P10War„  +  v,  +  sit 

an  x  percent  decrease  in  budget  can  be  interpreted  as  a  fix  increase  in  the  value  of  the 
cost  overruns  in  any  given  year.  For  example,  if  the  procurement  budget  were  to  drop  by 
1  percent,  this  model  predicts  that  cost  overruns  would  increase  in  value  by 
approximately  (-42.86)*(-0.01)  or  about  0.43  meaning  that  that  weapon  system  costs 
would  increase  by  0.43  percent.  Similarly,  if  the  Research  and  Development  (R&D) 
budget  decreased  by  10%,  cost  overruns  would  be  expected  to  increase  in  value  by 
approximately  -52.45*-0.1  or  -5.245  meaning  that  weapon  system  costs  would  be 
expected  to  increase  by  about  5.2%. 

To  clarify  this  second  example,  the  combined  R&D  budget  for  the  Air  Force, 
Army,  and  Navy  in  2002  was  $35.7  billion  so  a  10  percent  decrease  would  be  a  cut  in 
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budget  of  $3.57  billion.  Fixed-price  contract  cost  overruns  for  that  year  averaged 
approximately  4.2  percent  on  $19.4  billion  for  a  cost  overrun  in  that  year  of  about  $815 
million.  As  stated  above,  the  model  predicts  that  in  the  event  of  a  10%  cut  in  R&D 
budgets,  cost  overruns  would  increase  by  5.2  percent  to  9.4%  and  consequently  total 
overruns  would  increase  from  $815  million  to  $1.8  billion.  Said  another  way,  a  cut  of 
about  $3.5  billion  in  R&D  budgets  is  correlated  with  an  increase  in  cost  overruns  of  about 
$1  billion.  This  is  a  very  important  result  because  it  reveals  the  magnitude  and 
significance  of  funding  instability  on  cost  overruns  for  fixed-price  contracts. 

Model  2:  Contract  Cost  Overruns  (Cost-Plus  Contracts) 

The  dependent  variable  in  Model  2  is  the  average  annual  cost  overrun  per  service 
for  cost-plus  contracts.  Figure  6  is  a  visual  depiction  of  these  overruns. 


Cost  Overruns  (Cost-Plus  Contracts) 


■* -  Air  Force  - ■ -  Army 

♦  Navy 


Figure  6.  Cost  Overruns  of  Cost-Plus  Contracts  (1970-2002) 
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As  suggested  in  Chapter  III,  acquisition  reforms  and  budgets  are  expected  to  be 


most  significant  as  lagged  independent  variables.  The  lag  structure  for  Model  2  was 
determined  by  minimizing  the  Akaike  Information  Criterion  (AIC)  and  is  represented  in 
Table  23  below. 


Table  23,  Model  2  Lag  Structure 


Model  2  Lag  Structure  (Cost-Plus  Cost  Overruns) 

Variable 

Using  AIC 

Packard 

7 

FASA 

2 

DAWIA 

0 

Nunn-McCurdy 

2 

Budgets 

2 

From  the  regression  results  in  Table  25  below,  the  Baltagi-Wu  LBI  test  for  lag  1 
autocorrelation  in  model  2  has  a  value  of  1.45.  This  possibly  indicates  some  remaining 
autocorrelation  of  the  residuals.  However,  since  an  AR(1)  disturbance  term  is  already 
included  in  the  model,  no  additional  specification  changes  are  made. 

Using  the  Breusch-Pagan  test  shown  in  Table  24  below,  the  prob>chi2  is  greater 
than  a  = .  1 ,  therefore  we  fail  to  reject  the  null  hypothesis  of  constant  variance  and 
conclude  that  the  model  does  not  suffer  from  heteroskedasticity. 


Table  24.  Test  for  Constant  Variance  (Model  2) 


Model  2:  overrunpercentcplOO 


Breusch-Pagan  Test  (H0_ 

=Constant 

Variance) 

Estimated  Results 

Variance 

Standard  Deviation 

overmnpercent  1 00 

6.895362 

2.625902 

e 

3.620331 

1.902717 

u 

0 

0 

chi2(l) 

= 

0.45 

prob>chi2 

= 

0.5047 
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Using  the  AIC  as  the  lag  structure  criteria,  the  regression  results  are  as  follows: 


Table  25.  Model  2  Regression  Results 


Model  2:  Panel  Results  (Cost-Plus  Cost  Overruns) 

iFixed-effects  (within)  Regression  with  AR(1)  Disturbance 

Number  of  obs  = 

Group  variable  (i):  service 

Number  of  groups  = 

R-sq:  within  = 

0.4255 

Obs  per  group:  min  = 

22 

between  = 

0.1698 

avg  = 

22 

overall  = 

0.3924 

max  = 

22 

F(10,53) 

3.93 

Prob  >  F  = 

0.0005 

overrunpercentcplOO 

Coef. 

Std.  Err. 

t 

P>|t| 

O&M  Budget  (%Change)  (Lag  2) 

5.7208 

3.8505 

1.4900 

0.1430 

Procurement  Budget  (%  Change)  (Lag  2) 

4.6562 

4.1008 

1.1400 

0.2610 

R&D  Budget  (%  Change)  (Lag  2) 

9.7971 

4.8812 

2.0100 

0.0500 

Industry  Concentration  (CR4) 

41.8028 

31.1135 

-1.3400 

0.1850 

Unexpected  Inflation 

-0.0868 

0.0872 

-1.0000 

0.3240 

Packard  Commision  (Lag  7) 

-3.4107 

0.9626 

-3.5400 

0.0010 

FASA  (Lag  2) 

-2.2000 

0.8337 

-2.6400 

0.0110 

Nunn-McCurdy  Act  (Lag  2) 

-0.9894 

1.0894 

-0.9100 

0.3680 

DAWIA 

4.1948 

1.0319 

4.0700 

0.0000 

War 

-1.5748 

0.7680 

-2.0500 

0.0450 

_cons 

11.4387 

3.7272 

3.0700 

0.0030 

Baltagi-Wu  LBI 

1.4548 

The  results  of  this  analysis  show  R&D  budgets  and  O&M  budgets  are  positively 
correlated  with  cost  overruns  as  is  the  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA)  while  the  Packard  Commission  reforms,  the  Federal  Acquisition  Streamlining 
Act  (FASA),  and  war  are  negatively  correlated.  Table  26  compares  the  actual  impact  of 
each  independent  variable  on  cost  overruns  of  Cost-Plus  contracts  with  the  expected 
impact  from  past  research.  Independent  variables  that  are  not  significantly  correlated 
with  cost  overruns  are  left  blank  in  the  actual  impact  column. 
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Table  26.  Comparison  of  Expected  and  Actual  Impact  on  Cost  Overruns 


Model  2  (Cost-Plus  Cost  Overruns) 

Expected  Impact 

Actual 

INDEPENDENT  VARIABLES 

from  Past  Research 

Impact 

%AOperations  and  Maintenance  Budget 

+ 

%AProcurement  Budget 

- 

%  AResearch  and  Development  Budget 

- 

+ 

Industry  Concentration  (CR4) 

- 

Unexpected  Inflation 

+ 

Packard  Commission 

+ 

- 

FASA 

_ 

Nunn-McCurdy  Act 

DAWIA 

+ 

War 

+ 

- 

Notice  that  the  all  variables  in  this  model  were  either  not  significant  or  had  an 
impact  different  than  what  was  expected  from  past  research.  As  with  the  Fixed-Price 
model,  the  Packard  commission  reform  is  again  correlated  with  a  decrease  in  cost 
overruns.  This  is  again  contrary  to  the  findings  of  others  like  Searle  (1997),  Christensen 
et  al  (1999)  and  Holbrook  (2003).  Those  studies  either  concluded  that  the  Packard 
Commission  recommendations  were  correlated  with  an  increase  in  cost  overruns  or  they 
were  unable  to  show  any  significant  relationship.  Again,  the  difference  in  this  study  is 
the  inclusion  of  other  relevant  variables  in  the  model.  In  effect,  this  result  suggests  that 
the  Packard  Commission  and  many  other  acquisition  reforms  would  have  reduced  cost 
overruns  had  the  other  factors  not  overwhelmed  their  impact. 

One  major  difference  between  the  cost-plus  model  and  the  fixed-price  model  is 
that  in  this  model,  R&D  and  O&M  budgets  are  now  positively  correlated  with  cost 
overruns.  This  means  that  an  increase  in  R&D  or  O&M  budgets  would  result  in  an 
increase  in  cost-overruns  for  cost-plus  contracts.  This  is  counter  to  the  research  that 
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suggested  that  decreases  in  budgets  would  result  in  increases  in  cost  overruns.  It  is  worth 


noting  though  that  the  magnitude  of  the  coefficient  is  much  smaller  than  in  the  fixed- 
price  model  so  while  the  direction  is  unexpected,  the  predicted  impact  is  much  smaller. 


Table  27.  Model  2  Coefficients  and  P-Values 


Model  2  (Cost-Plus  Cost  Overruns) 

Within  R2= 

=0.4255 

INDEPENDENT  VARIABLES 

Coefficient 

Effi 

%AOperations  and  Maintenance  Budget 

5.720776 

0.143 

%AProcurement  Budget 

4.656242 

0.261 

%AResearch  and  Development  Budget 

9.797075 

0.05 

Industry  Concentration  (CR4) 

41.80284 

0.185 

Unexpected  Inflation 

-0.0868203 

0.324 

Packard  Commission 

-3.410674 

0.001 

FASA 

-2.199959 

0.011 

Nunn-McCurdy  Act 

-0.9894184 

0.368 

DAWIA 

4.19476 

0.000 

War 

-1.574791 

0.045 

Also,  it  is  interesting  to  note  that  for  the  second  straight  model,  industry 
concentration  is  not  significant.  As  discussed  in  the  literature  review,  DoD  expected 
some  cost  savings  from  consolidation.  However,  empirically  this  has  not  been  observed 
in  the  model  of  fixed-price  or  cost-plus  contracts.  In  fact,  with  a  p-value  of  .185  and  a 
large  positive  coefficient,  the  data  may  suggest  the  opposite  is  true.  That  is,  it  could  be 
that  the  increased  industry  consolidation  resulted  not  in  savings  but  increases  in  cost 
overruns  in  defense  contracts. 

This  leads  one  to  conclude  that  if  there  where  savings  from  consolidation  it  did 
not  come  from  a  decrease  in  the  cost  overruns  in  defense  contracts.  One  has  to  wonder,  if 
the  savings  are  not  in  the  contracts  then  were  they  actually  DoD  savings  or  just  savings  to 
the  defense  contractors  as  they  sized  their  businesses  to  fit  the  market  size  dictated  by  the 
post-cold  war  environment  of  decreasing  defense  budgets? 
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Also  worth  noting  is  that  acquisition  reforms  are  generally  significant  and  for  the 


most  part  negatively  correlated  with  cost  overruns.  This  is  an  important  finding  because 
past  research  on  acquisition  reform’s  impact  on  cost  overruns  was  inconclusive.  The 
only  reform  that  does  not  fit  this  pattern  is  the  Defense  Acquisition  Workforce 
Improvement  Act  which  is  again  positively  correlated  with  cost  overruns.  This  suggests 
that  cost  overruns  increased  after  DAWIA  was  enacted.  Perhaps  this  is  a  result  of  time 
away  from  the  job  for  training  or  perhaps  this  reveals  that  there  are  other  omitted 
variables  that  should  be  included  in  the  analysis. 

The  final  significant  variable  in  the  cost-plus  contract  model  is  war.  This  model 
reveals  that  war  is  negatively  correlated  with  cost  overruns.  This  suggests  that  the 
presence  of  war  reduces  contract  cost  overruns  of  cost-plus  contracts.  Perhaps  this  is 
because  some  weapon  system  programs  are  expedited  during  wartime.  The  urgency  of 
getting  the  system  out  the  door  precludes  the  desire  of  the  acquisitions  and  requirements 
communities  to  “gold  plate”  the  system  requirements.  It  could  be  that  that  last  amount  of 
added  capability  that  would  have  otherwise  increased  the  cost  is  not  included  as  a  result 
of  war. 

Model  3:  Contract  Cost  Overruns  (Production  Contracts) 

The  dependent  variable  form  Model  3  is  the  average  annual  cost  overrun  per 
service  for  Production  contracts.  Figure  7  is  a  visual  depiction  of  these  overruns. 
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Cost  Overruns  (Production  Contracts) 
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Figure  7.  Cost  Overruns  of  Production  Contracts  (1970-2002) 


As  suggested  in  Chapter  III,  acquisition  reforms  and  budgets  are  expected  to  be 
most  significant  as  lagged  independent  variables.  The  lag  structure  for  Model  3  was 
determined  by  minimizing  the  Akaike  Information  Criterion  (AIC).  For  more  on  how  the 
lags  were  determined  see  Appendix  D. 


Table  28.  Lag  Structure  (Model  3) 


Model  3  Lag  Structure  (Production  Cost-Overruns) 

Variable 

Using  AIC 

Packard 

4 

FASA 

1 

DAWIA 

0 

Nunn-McCurdy 

0 

Budgets 

3 
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From  the  regression  results  in  Table  30  below,  the  Baltagi-Wu  LBI  test  for  lag  1 
autocorrelation  in  the  specified  model  has  a  value  of  1.18.  This  appears  to  indicate  some 
remaining  autocorrelation  of  the  residuals.  However,  since  an  AR(1)  disturbance  term  is 
already  included  in  the  model,  no  additional  specification  changes  are  made. 

Using  the  Breusch-Pagan  test  from  Table  29,  the  prob>chi2  is  greater  than  a  -  A, 
therefore  we  fail  to  reject  the  null  hypothesis  of  constant  variance  and  conclude  that  the 
model  does  not  suffer  from  heteroskedasticity. 


Table  29.  Test  for  Constant  Variance  (Model  3) 


Model  3: 

overrunpercentproclOO  (min  AIC) 

Breusch-Pagan  Test  (H0=Constant 

Variance) 

Estimated  Results 

Variance  Standard  Deviation 

overmnpercent  1 00 

22.82678 

4.777738 

e 

18.2617 

4.273371 

u 

0 

0 

chi2(l) 

= 

0.02 

prot»chi2 

= 

0.8893 

Using  the  lag  structure  from  Table  28  above,  the  regression  results  are  as  follows: 
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Table  30.  Regression  Results  for  Model  3:  Production  Cost  Overruns 


Model  3:  Panel  Results  (Production  Cost  Overruns) 

iFixed-effects  (within)  Regression  with  AR(1)  Disturbance 

Number  of  obs  = 

Group  variable  (i):  service 

Number  of  groups  = 

R-sq: 

within  = 

0.3873 

Obs  per  group:  min  = 

21 

between  = 

0.1069 

avg  = 

21.7 

overall  = 

0.4191 

max  = 

22 

F(10,52) 

3.29 

Prob  >  F  = 

0.0023 

overrunpercentproclOO 

Coef. 

Std.  Err. 

t 

P>|t| 

O&M  Budget  (%Change)  (Lag  3) 

-5.0048 

6.4718 

-0.7700 

0.4430 

Procurement  Budget  (%  Change)  (Lag  3) 

-6.1097 

6.8092 

-0.9000 

0.3740 

R&D  Budget  (%  Change)  (Lag  3) 

-7.3763 

8.1314 

-0.9100 

0.3690 

Industry  Concentration  (CR4) 

0.0372 

0.1799 

0.2100 

0.8370 

Unexpected  Inflation 

4.8517 

49.0284 

0.1000 

0.9220 

Packard  Commision  (Lag  4) 

2.6483 

1.8005 

1.4700 

0.1470 

FASA  (Lag  1) 

-3.3674 

1.8021 

-1.8700 

0.0670 

Nunn-McCurdy  Act 

-8.4624 

2.2343 

-3.7900 

0.0000 

DAWIA 

0.5980 

2.4087 

0.2500 

0.8050 

War 

1.9695 

1.0019 

1.9700 

0.0550 

_cons 

9.3682 

5.2320 

1.7900 

0.0790 

Baltagi-Wu  LBI 

1.1934 

The  results  of  this  model  reveal  that  the  Federal  Acquisition  Streamlining  Act 
(FASA),  and  the  Nunn-McCurdy  Act  are  negatively  correlated  with  cost  overruns  in 
production  contracts  while  the  Packard  Commission  reforms  and  war  are  positively 
correlated.  Table  31  compares  the  actual  impact  of  each  independent  variable  on  cost 
overruns  of  Production  contracts  with  the  expected  impact  from  past  research. 
Independent  variables  that  are  not  significantly  correlated  with  cost  overruns  are  left 
blank  in  the  actual  impact  column. 
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Table  31.  Comparison  of  Expected  and  Actual  Impact  on  Cost  Overruns 


Model  3  (Production  Cost  Overruns) 

Expected  Impact 

Actual 

INDEPENDENT  VARIABLES 

from  Past  Research 

Impact 

%  A  Operations  and  Maintenance  Budget 

%AProcurement  Budget 

_ 

%  AResearch  and  Development  Budget 

- 

Industry  Concentration  (CR4) 

- 

Unexpected  Inflation 

+ 

Packard  Commission 

+ 

+ 

FASA 

- 

Nunn-McCurdy  Act 

DAWIA 

- 

War 

+ 

+ 

Unlike  the  models  of  contract  type,  in  this  model  of  production  contracts,  budgets 
are  not  significantly  correlated  with  cost  overruns.  Also,  as  with  the  previous  models, 
industry  concentration  is  not  significantly  correlated  with  cost  overruns  or  cost  savings 
like  the  DoD  expected  (GAO,  1998:2). 


Table  32.  Model  3  Coefficients  and  P-Values 


Model  3  (Production  Cost  Overruns) 

within  R2= 

=0.3873 

INDEPENDENT  VARIABLES 

Coefficient 

p-value 

%AOperations  and  Maintenance  Budget 

-5.0048 

%AProcurement  Budget 

-6.1097 

%AResearch  and  Development  Budget 

-7.3763 

Industry  Concentration  (CR4) 

0.0372 

0.8370 

Unexpected  Inflation 

4.8517 

0.9220 

Packard  Commission 

2.6483 

0.1470 

FASA 

-3.3674 

0.0670 

Nunn-McCurdy  Act 

-8.4624 

0.0000 

DAWIA 

0.5980 

0.8050 

War 

1.9695 

0.0550 

Also,  in  this  model,  the  Packard  Commission  recommendations  have  a  p-value  of  .147 
and  a  positive  correlation  with  cost  overruns.  While  only  bordering  on  statistical 
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significance  and  small  in  magnitude,  this  result  is  consistent  with  research  by  Searle 
(1997)  and  Christensen  et  al  (1999). 

Further,  a  trend  appears  to  be  emerging  that  indicates  the  Federal  Acquisition 
Streamlining  Act  (FASA)  and  the  Nunn-McCurdy  Act  are  correlated  with  decreases  in 
cost  overruns.  Since  the  Nunn-McCurdy  act  was  enacted  with  the  intent  to  control 
defense  weapon  system  cost  overruns  it  reassuring  to  note  that  it  is  passage  appears  to  be 
correlated  with  a  decrease  in  cost  overrun  though  admittedly  the  magnitude  is  small. 

The  final  significant  variable  in  the  production  model  is  war.  In  this  model,  war 
is  positively  correlated  with  an  increase  in  cost  overruns.  This  contrasts  with  the  result  of 
the  cost-plus  contract  model.  A  possible  explanation  for  this  is  that  products  are  rushed 
from  R&D  into  production  in  times  of  war  at  an  increased  production  cost. 

Model  4.  Contract  Cost  Overruns  (Research  and  Development  Contracts) 

The  dependent  variable  in  the  R&D  model  is  the  average  annual  cost  overrun  per 
service  for  R&D  contracts.  Figure  8  is  a  visual  depiction  of  these  overruns. 
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Cost  Overruns  (R&D  Contracts) 


Figure  8.  Cost  Overruns  of  R&D  Contracts  (1970-2002) 


As  suggested  in  Chapter  III,  acquisition  reforms  and  budgets  are  expected  to  be 
most  significant  as  lagged  independent  variables.  The  lag  structure  for  model  1  was 
determined  by  minimizing  the  Akaike  Information  Criterion  (AIC).  For  more  on  how  the 
lags  were  determined  see  Appendix  D. 


Table  33.  Lag  Structure  (Model  4) 


Model  4  Lag  Structure  (R&D  Cost  Overruns) 

Variable 

Using  AIC 

Packard 

3 

FASA 

1 

DAWIA 

0 

Nunn-McCurdy 

3 

Budgets 

0 
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From  the  regression  results  in  Table  34  below,  the  Baltagi-Wu  LBI  test  for  lag  1 


autocorrelation  in  the  specified  model  has  a  value  of  .598.  This  appears  to  indicate 


significant  remaining  autocorrelation  of  the  residuals. 


Table  34.  Regression  Results  for  R&D  Contract  Model  Before  Lags 


Model  4:  Panel  Results  (R&D  Cost  Overruns) 

iFixed-effects  (within)  Regression  with  AR(1)  Disturbance 

Number  of  obs  = 

Group  variable  (i):  service 

Number  of  groups  = 

3 

R-sq:  within  = 

0.3372 

Obs  per  group:  min  = 

22 

between  = 

0.1297 

avg  = 

22 

overall  = 

0.0695 

max  = 

22 

F(10,53) 

2.7 

Prob  >  F  = 

0.0095 

overrunpercentrdl  00 

Coef. 

Std.  Err. 

t 

P>|t| 

O&M  Budget  (%Change) 

-12.8577 

6.5148 

-1.9700 

0.0540 

Procurement  Budget  (%  Change) 

-13.2747 

6.9110 

-1.9200 

0.0600 

R&D  Budget  (%  Change) 

-11.4385 

8.2788 

-1.3800 

0.1730 

Industry  Concentration  (CR4) 

-0.3274 

0.4640 

-0.7100 

0.4830 

Unexpected  Inflation 

45.0504 

44.6357 

1.0100 

0.3170 

Packard  Commision  (Lag  3) 

2.1285 

1.4304 

1.4900 

0.1430 

FAS  A  (Lag  1) 

-5.1298 

1.9871 

-2.5800 

0.0130 

Nunn-McCurdy  Act  (Lag  3) 

-1.7796 

2.1933 

-0.8100 

0.4210 

DAWIA 

6.1415 

1.9696 

3.1200 

0.0030 

War 

0.7912 

0.9122 

0.8700 

0.3900 

cons 

31.3844 

5.6378 

5.5700 

0.0000 

Baltagi-Wu  LBI 

0.5975 

Also,  from  the  p-value  of  the  Breusch-Pagan  test,  it  is  clear  that  this  model  suffers 
from  heteroskedasticity  or  non-constant  variance  of  the  error  terms.  This  makes  the 
coefficients  of  the  panel  model  less  efficient. 


Table  35.  Test  for  Constant  Variance  (Model  4) 


Model  4:  overrunpercentrdlOO  (min  AIC) 


Breusch-Pagan  Test  (H0=Constant  Variance) 


Estimated  Results 

overmnpercent  1 00 
e 

u 

chi2(l) 

prob>chi2 


Variance _ 

51.91042 

26.03706 

0 


Standard 

Deviation 

7.204889 

5.102652 

0 

7.34 

0.0067 
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To  account  for  the  autocorrelation,  the  first  two  lagged  dependent  variables  are 


included  as  independent  variables.  With  this  change  in  model  specification,  the  Baltagi- 
Wu  LBI  increases  to  2.07  indicating  that  the  autocorrelation  is  sufficiently  accounted  for. 
Table  35  is  a  summary  of  the  new  regression  results  for  Model  4  with  the  lagged 
dependent  variables  included. 


Table  36.  Regression  Results  for  R&D  Contract  Model  with  Lags 


Model  4:  Panel  Results  (R&D  Cost  Overruns) 

1  Fixed-effects  (within)  Regression  with  AR(1)  Disturbance 

Number  of  obs  = 

66 

Group  variable  (i):  service 

Number  of  groups  = 

3 

K-sq:  within  = 

0.7711 

Obs  per  group:  min  = 

22 

between  = 

0.8059 

avg  = 

22 

overall  = 

0.8353 

max  = 

22 

F(  12,5 1) 

14.32 

Prob  >  F  = 

0.0000 

overrunpercentrdl  00 

Coef. 

Std.  Err. 

t 

P>|t| 

overrunpercentrdlOO  Lag  1 

0.8776 

0.1447 

overrunpercentrdl 00  Lag  2 

-0.2647 

0.1390 

-1.9000 

0.0630 

O&M  Budget  (%Change) 

-16.9281 

6.4919 

-2.6100 

0.0120 

Procurement  Budget  (%  Change) 

-17.4210 

7.1730 

-2.4300 

0.0190 

R&D  Budget  (%  Change) 

-18.0752 

8.4405 

-2.1400 

0.0370 

Industry  Concentration  (CR4) 

-0.0287 

0.1377 

-0.2100 

0.8360 

Unexpected  Inflation 

-55.6643 

54.1644 

-1.0300 

0.3090 

Packard  Commision  (Lag  3) 

0.0614 

1.3637 

0.0500 

0.9640 

FASA  (Lag  1) 

-3.8548 

1.7174 

-2.2400 

0.0290 

Nunn-McCurdy  Act  (Lag  3) 

-0.9449 

1.9207 

-0.4900 

0.6250 

DAWIA 

2.9565 

1.7072 

1.7300 

0.0890 

War 

0.0938 

1.0970 

0.0900 

0.9320 

cons 

3.2714 

6.6569 

0.4900 

0.6250 

Baltagi-Wu  LBI 

2.0787 

Below  is  a  table  depicting  the  expected  impact  from  past  research  of  each 
independent  variable  on  cost  overruns  and  the  actual  impact  for  the  R&D  cost  overrun 
model. 
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Table  37.  Comparison  of  Expected  and  Actual  Impact  on  Cost  Overruns 


Model  4  (R&D  Cost  Overruns) 

Expected  Impact 

Actual 

INDEPENDENT  VARIABLES 

from  Past  Research 

Impact 

%  A  Operations  and  Maintenance  Budget 

%AProcurement  Budget 

- 

- 

%  AResearch  and  Development  Budget 

- 

- 

Industry  Concentration  (CR4) 

- 

Unexpected  Inflation 

+ 

Packard  Commission 

+ 

FASA 

_ 

Nunn-McCurdy  Act 

DAWIA 

+ 

War 

+ 

In  this  model,  O&M  and  Procurement  budgets  are  negatively  correlated  with  cost 
overruns  of  R&D  contracts.  This  implies  that  all  else  being  equal,  an  increase  in  defense 
budgets  would  result  in  a  decrease  in  cost  overruns  and  a  decrease  in  defense  budgets 
would  result  in  an  increase  in  cost  overruns.  This  finding  is  consistent  with  past  research 
by  Singleton  (1991),  and  Drezner  et  al.  (1996)  and  this  is  the  second  model  of  this 
research  where  this  relationship  has  appeared. 

Notice  from  Table  38  that  as  with  all  previous  models,  the  Federal  Acquisition 
Streamlining  Act  (FAS A)  is  negatively  correlated  with  cost  overruns.  To  understand  the 
magnitude  of  this  result,  consider  that  the  FASA  reforms  are  correlated  with  more  than  a 
5%  reduction  in  the  annual  cost  overrun  percentage.  So,  for  example,  in  1993,  the  year 
before  FASA  was  enacted,  the  R&D  budget  was  just  over  $32  billion  and  cost  overruns 
in  that  year  averaged  16.8  percent.  According  to  this  model,  all  else  being  equal,  having 
the  FASA  reforms  in  1993  would  have  resulted  in  only  an  1 1.7  percent  cost  overrun  for  a 
savings  in  that  year  of  over  $1.6  billion. 
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Table  38.  Model  4  Coefficients  and  P-Values 


Model  4  (R&D  Cost  Overruns) 

within  R2= 

=0.3372 

INDEPENDENT  VARIABLES 

Coefficient 

p-value 

%AOperations  and  Maintenance  Budget 

-12.8577 

0.0540 

%AProcurement  Budget 

-13.2747 

0.0600 

%AResearch  and  Development  Budget 

-11.4385 

0.1730 

Industry  Concentration  (CR4) 

-0.3274 

0.4830 

Unexpected  Inflation 

45.0504 

0.3170 

Packard  Commission 

2.1285 

0.1430 

FASA 

-5.1298 

0.0130 

Nunn-McCurdy  Act 

-1.7796 

0.4210 

DAWIA 

6.1415 

0.0030 

War 

0.7912 

0.3900 

Chapter  Summary 

This  chapter  presented  the  results  of  four  panel  models  created  to  explain  cost 
overruns  in  defense  weapon  system  contracts.  Chapter  V  continues  the  discussion  of 
these  findings  and  their  implications  as  well  as  presents  recommendations  for  further 
research. 
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V.  Conclusions  and  Recommendations 


Review  of  Research  Objectives 

The  primary  purpose  of  this  research  was  to  create  an  empirical  model  to  explain 
cost  overruns  in  defense  weapon  system  programs.  This  was  accomplished  using  four 
fixed-effects  panel  models.  Recall  that  prior  to  this  study  no  empirical  model  existed  to 
examine  the  relative  impact  of  factors  contributing  to  cost  overruns  in  a  dynamic  setting. 
Past  research  had  instead  focused  on  qualitative  explanations  or  hypothesis  tests  which 
are  limited  by  a  subjective  treatment  date  and  biased  by  omitted  variables. 

To  create  the  panel  model,  data  was  gathered  from  the  DAES  database  on  the  cost 
overruns  of  186  major  weapon  system  programs  of  the  Air  Force,  Army,  and  Navy  from 
1970  to  2002.  The  cost  data  was  then  separated  into  overruns  by  contract  type  and 
program  phase.  Since  past  research  identified  funding  instability,  industry  consolidation, 
estimation  error,  and  acquisition  refonn  as  potentially  impacting  cost  overruns,  data  on 
these  independent  variables  was  compiled  and  used  to  create  the  panel  model.  In  this 
research,  the  annual  change  in  budgets  was  a  proxy  for  funding  instability  while  the  CR4 
measured  industry  consolidation.  Estimation  error  was  modeled  as  the  amount  of 
unexpected  inflation  in  the  initial  estimate  and  acquisition  reforms  and  war  were  modeled 
as  dummy  variables. 
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Table  39.  Independent  Variables  in  the  Panel  Model 


Factor 

Model  Variable 

Expected  Impact 

Funding  Instability 

Percentage  Change  in  Defense  Budgets 

+ 

Industry  Consolidation 

Industry  Concentration  (CR4) 

- 

War 

War  Dummy  (Vietnam,  Gulf  War,  Bosnia,  Post- 
September  1 1th) 

+ 

Cost  Estimating  Error 

Unexpected  Inflation  in  DoD 

+ 

Acquisition  Reform 

Dummy  Variables  for  Nunn-McCurdy,  Packard, 
FASA,  DAWIA 

Inconclusive 

The  study  proposed  four  research  questions: 

1.  Did  the  defense  industry  consolidation  and  concurrent  decrease  in  defense 
budgets  that  occurred  in  the  1990’s  affect  the  cost  of  defense  weapon  systems? 

2.  Is  war  correlated  with  an  increase  in  weapon  system  cost  overruns? 

3.  Is  estimation  error  caused  by  unexpected  inflation  correlated  with  an  increase  in 
weapon  system  cost  overruns? 

4.  Did  acquisition  reforms  have  impact  on  cost  overruns  when  defense  industry 
consolidation,  inflation,  changes  in  the  defense  budget,  and  war  are  considered? 


Discussion  of  Results 

Table  40  is  a  summary  of  the  regression  results  from  the  four  models  and  is 
referenced  throughout  this  chapter  as  each  research  question  is  discussed. 
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Table  40.  Summary  of  Model  Results 


Variable 

Fixed- 

Price 

Cost-Plus 

Production 

R&D 

O&M  Budget  (%Change) 

-31.50*** 

5.72* 

-5.00 

-16.93*** 

Procurement  Budget  (%  Change) 

-42.86*** 

4.66 

-6.11 

-17.42*** 

R&D  Budget  (%  Change) 

-52.45*** 

9.80** 

-7.38 

-18.08*** 

Industry  Concentration  (CR4) 

0.15 

41.80 

0.04 

-0.03 

Unexpected  Inflation 

134.00 

-0.09 

4.85 

-55.66 

Packard  Commission 

-6.35* 

3  4  ■'I  *** 

2.65 

0.06 

FASA 

-22.00*** 

-2.20** 

-3.37* 

-3.85*** 

Nunn-McCurdy  Act 

-9.10** 

-0.99 

-8.46*** 

-0.94 

DAWIA 

13.09*** 

4.19*** 

0.60 

2.96* 

War 

1.29 

-1.57** 

1.97* 

0.094 

Defense  Industry  Consolidation 

Despite  the  Department  of  Defense  expectation  that  the  defense  industry 
consolidation  of  the  1990’s  would  result  in  cost  savings  (GAO,  1998: 1),  this  research  was 
unable  to  find  any  evidence  of  that  savings.  A  quick  glance  at  Table  40  and  it  is  clear 
that  industry  consolidation  was  not  significant  regardless  of  program  phase  or  contract 
type.  If  anything,  industry  concentration  may  have  resulted  in  increased  cost  to  DoD 
especially  in  cost-plus  contracts  with  a  coefficient  of  41.8  and  a  p-value  of  .18  which  is 
right  outside  the  traditional  level  of  statistical  significance  but  significant  in  magnitude. 


Defense  Budgets 

In  this  research,  defense  budgets  were  a  proxy  for  funding  instability.  Past 
research  suggested  that  funding  instability  was  negatively  correlated  with  cost  overruns. 
In  other  words,  decreases  in  defense  budgets  were  expected  to  result  in  increases  in  cost 
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overruns.  This  result  held  in  the  Fixed  Price  and  Research  and  Development  models  and 
was  highly  statistically  significant. 

To  illustrate  the  magnitude  of  this  finding,  if  the  2002  R&D  budget  were  to  drop 
by  1  percent  in  the  fixed  price  model,  cost  overruns  would  be  expected  to  increase  by 
approximately  (-52. 45*-. 01)  or  0.52  percent.  At  first  this  does  not  sound  significant  but 
consider  that  cost  overruns  in  that  year  averaged  approximately  4.2  percent  on  $19.4 
billion  for  a  cost  overrun  in  that  year  of  about  $815  million.  An  increase  in  cost  overruns 
of  a  half  a  percent  to  4.7%  for  the  year  would  result  in  an  additional  $96  million  in 
overruns.  In  other  words,  for  every  dollar  of  R&D  budget  cuts,  overruns  increase  by  50 
cents. 

Why  might  funding  instability  result  be  so  strongly  correlated  with  increases  in 
cost  overruns?  One  possible  answer  is  that  when  budgets  decrease,  existing  programs 
engage  in  a  fight  for  available  funding  and  implicitly,  the  lower  priority  programs  get 
their  funding  cut.  But,  the  programs  are  not  cancelled.  Instead,  they  just  continue  on  with 
inadequate  funding  that  causes  schedule  delays  and  increased  cost  due  to  production 
breaks  and  orphaned  technology.  In  the  final  analysis,  perhaps  the  funding  instability  is 
so  highly  correlated  with  increases  in  cost  overruns  because  of  poor  portfolio 
management  by  the  DoD  during  times  of  decreasing  budgets.  This  idea  that  the  DoD 
“corporate  process”  for  acquisitions  exacerbates  cost  and  schedule  delays  is  one  first 
proposed  by  McNutt  (1998)  but  not  previously  empirically  quantified.  What  McNutt 
meant  by  portfolio  management  is  that  the  DoD  essentially  has  a  portfolio  of  weapon 
systems  in  development  that  they  manage  much  like  a  car  company  would  manage  their 
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line  of  products.  In  the  case  of  DoD,  he  suggested  that  the  Air  Force  and  by  extension 
DoD  does  a  poor  job  managing  their  product  line  essentially  in  times  of  decreasing 
budgets.  . 

War 

Past  research  by  Czelusniak  and  Rogers  (1997)  suggested  that  funding  instability 
from  contingency  operations  leads  to  cost  growth.  This  study  attempted  to  replicate  that 
finding  by  modeling  war  with  a  dummy  variable  in  the  panel  model.  Model  results  reveal 
that  war  is  positively  correlated  with  cost  overruns  in  Cost-Plus  contract  types  and 
negatively  correlated  with  cost  overruns  in  the  Production  phase. 

Why  might  this  be?  At  the  onset  of  any  war,  service  chiefs  call  on  the  acquisition 
community  to  provide  any  increase  in  weapon  system  capability  that  they  can  quickly 
push  out  to  the  battlefield.  These  programs  are  likely  to  be  in  the  R&D  phase  and  using 
cost-plus  contracts  because  of  their  increased  risk.  Products  nearing  the  end  of  R&D  are 
likely  moved  quickly  out  of  R&D  and  into  production,  skipping  any  “gold  plating”  of 
requirements  that  would  have  otherwise  been  done.  In  this  scenario,  it  is  reasonable  to 
expect  cost-plus  contracts  to  decrease  during  war  time.  Then,  those  same  products  would 
be  rushed  into  production  intuitively  causing  production  costs  to  increase  during  a  war. 

Cost  Estimation  Error 

Error  in  the  cost  estimate  is  identified  as  a  causal  factor  in  cost  overruns  by 
Christensen  et  al.  (1999).  This  research  modeled  cost  estimation  error  as  unexpected 
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inflation  in  the  cost  estimate.  All  cost  estimates  and  budgets  include  assumptions  about 
future  inflation.  These  inflation  predictions,  prepared  annually  by  the  Undersecretary  of 
Defense,  Comptroller  are  not  always  accurate.  For  example,  in  the  early  1980’s  inflation 
was  very  high  but  DoD  predictions  did  not  anticipate  this.  Consequently,  inflation  was 
underestimated. 

This  research  included  this  in  the  models  of  cost  overruns.  However,  the  results 
shown  in  Table  40  show  that  unexpected  inflation  is  not  significantly  correlated  with  cost 
overruns.  The  implication  of  this  finding  is  that  while  cost  estimating  error  may 
contribute  to  cost  overruns,  it  is  not  as  a  result  of  errors  in  the  inflation  predictions.  It  is 
worth  noting  however,  that  the  coefficient  in  the  fixed-price  model  is  very  high  and  just 
outside  the  range  where  it  would  traditionally  be  considered  significant.  Perhaps  if 
inflation  data  from  the  1970’s  were  able  to  be  obtained  and  included  in  future  research,  it 
would  prove  to  be  significant. 

Acquisition  Reforms 

The  fourth  question  asked  if  acquisition  refonns  had  an  impact  on  cost  overruns 
when  defense  industry  consolidation,  inflation,  changes  in  the  defense  budget,  and  war 
were  considered.  Specifically,  this  study  surveyed  past  research  on  acquisition  reform 
and  identified  four  reforms  as  being  most  significant.  Table  41  list  the  reforms  modeled 
in  this  study. 
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Table  41.  Acquisition  Reforms  Studied 

Acquisition  Reforms  Studied 


Nunn-McCurdy  Act  of  1 982 
Packard  Commission  Recommendations  of  1986 
The  Defense  Acquisition  Workforce  Improvement  Act  of  1990 
The  Federal  Acquisition  Streamlining  Act  of  1994 


Past  research  on  acquisition  refonn  has  been  unable  to  show  any  improvement  as 
a  result  of  reform.  This  idea  is  summed  up  be  Christensen  et  al  (1999)  when  they  said 
“Despite  the  implementation  of  more  than  two  dozen  regulatory  and  administrative 
initiatives,  there  has  been  no  substantial  improvement  in  the  cost  perfonnance  of  defense 
programs  for  more  than  30  years”  (Christensen  et  al,  1999:252). 

However,  as  discussed  in  earlier  chapters,  past  research  on  the  impact  of 
acquisition  reform  suffered  from  the  use  of  subjective  treatment  dates  and  omission  of 
variables  that  biased  their  results.  Additionally,  the  techniques  such  as  qualitative  case 
studies  and  hypothesis  tests  limited  the  robustness  of  past  results.  Indeed,  once  the  other 
factors  mentioned  above  were  modeled,  all  four  of  the  reforms  studied  were  correlated 
with  cost  overruns  in  at  least  one  model. 

Nunn-McCurdy  Act  of  1982 

The  Nunn-McCurdy  Act  of  1982  was  significantly  correlated  with  a  decrease  in 
cost  overruns  in  Fixed-Price  contracts  and  Production  contracts.  Recall  that  the  purpose 
of  the  Nunn-McCurdy  Act  was  to  curtail  cost  overruns  in  defense  weapon  systems.  Yet, 
over  20  years  after  its  enactment,  there  had  been  little  research  studying  its  impact.  To 
understand  the  magnitude  of  this  finding,  consider  that  model  results  suggest  that  all  else 
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being  equal,  the  added  presence  of  the  Nunn-McCurdy  act  lowered  contract  cost  overruns 
by  9. 1  percent  in  Fixed  Price  contracts  and  by  8.5  percent  in  Production  contracts. 

The  Packard  Commission  (1986) 

One  of  the  most  researched  acquisition  reforms  was  the  Packard  Commission 
recommendations  of  1986.  This  purpose  of  this  reform  was  to  improve  the  defense 
acquisition  process  by  streamlining  the  chain  of  command,  improving  tests  and 
prototyping  and  planning,  and  modeling  the  DoD  after  a  competitive  firm  (Searle, 
1997:33). 

The  idea  was  that  these  changes  would  result  in  weapon  systems  that  could  be 
produced  more  quickly  and  at  a  lower  cost.  Yet,  past  research  by  Searle  (1997)  and 
Christensen  et  al.  (1999)  suggested  that  the  Packard  Commission  recommendations  had 
no  impact  on  cost  overruns  and  that  the  overruns  actually  got  worse  after  the  Packard 
Commission  reforms  were  implemented. 

This  research  contradicts  these  findings  somewhat.  In  two  of  the  four  models,  fixed 
price  and  cost-plus  cost  overruns,  the  Packard  Commission  recommendations  are 
negatively  correlated  with  contracted  cost  overruns.  In  other  words,  the  presence  of  the 
Packard  Commission  reforms  was  correlated  with  a  decrease  in  contract  cost  overruns. 
Especially  significant  is  the  result  in  the  Cost-Plus  contract  model  which  illustrated  that 
all  else  being  equal,  the  presence  of  the  Packard  Commission  reforms  resulted  in  a  3.4 
percent  decrease  in  cost  overruns.  This  result  is  significant  to  the  cc=. 01  level. 
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DAWIA  (1990) 

In  1990,  the  Defense  Acquisition  Workforce  Improvement  Act  was  enacted  to 
improve  the  level  of  education  of  the  acquisition  workforce.  Intuitively,  one  would 
expect  that  a  more  well  trained  and  educated  workforce  would  be  able  to  deliver  weapon 
systems  at  a  decreased  cost.  However,  results  of  this  analysis  show  that  DAWIA  is 
positively  correlated  with  cost  overruns.  The  cause  of  this  result  is  unknown.  Perhaps 
the  finding  suggests  that  the  time  away  from  work  that  is  instead  spent  on  training  is 
causing  cost  performance  to  suffer.  Or,  perhaps  this  correlation  is  just  contemporaneous 
meaning  that  it  is  statistically  significant  but  not  as  a  result  of  any  causal  link. 

FAS  A  ( 1994) 

In  1994,  the  Federal  Acquisition  Streamlining  Act  was  created  to  empower  the 
acquisition  workforce  to  make  decisions  about  how  to  manage  their  programs.  This  was 
a  movement  away  from  the  restrictive  and  centralized  procurement  practices  of  the  past. 
(Cooper,  2002: 15).  Other  key  provision  of  FASA  were  the  increased  use  of 
performance  based  contract  payments,  and  the  push  towards  more  “commercial  or  off  the 
shelf  products”  (Cooper,  2002:18).  Most  important  for  this  study  were  the  requirements 
that  contractors  have  a  certified  cost  accounting  system  that  complies  with  accepted  cost 
accounting  standards  and  also  the  increased  scrutiny  of  a  contractor’s  past  perfonnance  in 
the  source  selection  criterion  of  major  weapon  system  purchases.  Clearly  these 
provisions  should  have  led  to  decreases  in  weapon  system  cost  and  should  be 
investigated.  Indeed,  in  their  1999  paper,  Christensen  et  al.  suggest  that  the  impact  of  the 
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FASA  reform  is  worthy  of  study  but  that  some  time  would  have  to  pass  before  its  impact 
could  be  felt  (Christensen  et  al,  1999:  258). 

Now,  twelve  years  after  the  reform,  the  impact  was  investigated  by  this  research 
and  indeed  FASA  did  reduced  cost  overruns.  This  result  held  in  all  models  regardless  of 
contract  type  or  program  phase  and  all  else  being  equal,  the  presence  of  FASA  reduced 
cost  overruns  by  as  little  as  2%  in  Cost-Plus  contracts  to  as  much  as  21%  in  Fixed-Price 
contracts. 

Conclusions 

The  purpose  of  this  research  was  to  create  an  empirical  model  that  described  cost 
overruns  in  defense  weapon  systems.  This  was  accomplished  by  modeling  cost  overruns 
of  186  major  weapons  system  programs  of  the  Air  Force,  Army,  and  Navy  over  a  period 
of  32  years. 

Major  findings  of  this  research  are  as  follows: 

•  The  Defense  Industry  Consolidation  of  the  1990’s  did  not  result  in  significant  cost 
savings  for  the  Department  of  Defense  and  might  even  have  resulted  in  increases 
in  cost  overruns. 

•  Funding  Instability  is  highly  correlated  with  an  increase  in  cost  overruns  in  Fixed- 
Price  contracts  and  contracts  in  the  Research  and  Development  Phase.  This  could 
be  a  result  of  poor  portfolio  management  by  the  DoD  in  times  of  decreasing 
budgets.  Funding  instability  contributed  to  an  increase  in  contract  cost  overruns 
of  $13.4  billion  since  1970. 
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•  War  is  correlated  with  decreases  in  Cost-Plus  contract  cost  overruns  and  increases 


in  Production  cost  overruns.  This  could  illustrate  the  impact  of  nearly  completed 
weapons  modifications  being  pushed  into  production  early  at  the  onset  of  war. 

•  Estimation  error  caused  by  unexpected  inflation  is  not  significantly  correlated 
with  cost  overruns.  If  estimation  error  in  cost  estimates  is  a  causal  factor  in 
contract  cost  overruns,  it  is  likely  the  result  of  some  other  aspect  of  weapon 
system  cost  estimating. 

•  With  the  exception  of  DAWIA,  acquisition  refonns  are  correlated  with  a  decrease 
in  cost  overruns  of  defense  weapon  systems.  In  particular,  reforms  resulting  from 
the  Nunn-McCurdy  Act  of  1982,  the  Packard  Commission  Recommendations  of 
1986  and  the  Federal  Acquisition  Streamlining  Act  (FASA)  of  1994  resulted  in 
savings  of  almost  $124  billion  since  1982.  Yet,  the  biggest  implication  of  this 
finding  is  not  the  correlation  of  one  specific  reform  with  contract  cost  overruns 
but  the  consistency  with  which  the  acquisition  refonns  did  matter  and  did  act  to 
reduce  cost  overruns  in  defense  weapon  systems. 
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Recommendations  for  Future  Research 


This  research  has  created  an  empirical  model  that  has  laid  the  foundation  for 
further  study  of  the  causes  of  cost  overruns  in  defense  weapon  systems.  Without  a 
doubt,  there  are  other  factors  related  to  weapon  system  costs  that  were  not  modeled  in 
this  study.  For  example,  in  his  confirmation  hearing,  Air  Force  Chief  of  Staff,  General 
Michael  Mosely  said  that  acquisition  corps  manning  decreases  may  have  contributed  to 
weapon  system  cost  growth.  That  data  could  easily  be  added  to  the  model  created  in  this 
thesis.  Also,  other  importance  events  such  as  the  formation  of  the  Air  Force  Materiel 
Command  in  the  early  1990’s  could  be  modeled.  Finally,  the  research  linking  cost 
overruns  and  schedule  delay  is  inconclusive.  A  model  that  incorporated  schedule  as  an 
independent  variable  explaining  cost  overruns  or  a  model  that  treated  schedule  as  the 
dependent  variable  and  cost  overruns  as  an  independent  variable  would  also  advance  the 
current  level  of  research. 
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Appendix  A.  Summary  of  DAES  Data 


Air  Force 


Program  Name 

Number  of  Contract  Entries 

A-10 

38 

ACM 

82 

Advanced  Extremely  High  Frequency  Satellite  (AEHF) 

10 

AFATDS 

2 

Airborne  Laser  (ABL) 

23 

ALCM 

91 

AMRAAM  (AIM-120A) 

175 

ASAT 

83 

ATS 

29 

B-1  CMUP-DSUP 

2 

B-1B 

432 

B-1B  CMUP 

62 

B-2A 

15 

C-130  Avionics  Modernization  Program  ((C-130  AMP) 

2 

C-17A 

291 

CMU 

115 

CSRL 

23 

DMSP 

176 

DSCS  III  A&B 

65 

DSP 

294 

E-3  AW  ACS  RSIP 

73 

E-3A  Hawkeye 

125 

E-4  (AABNCP) 

34 

EF-111A 

35 

EJS 

13 

F/A-22  raptor 

91 

F-15 

221 

F-16 

270 

GBS 

17 

GLCM 

58 

Global  Hawk  Unmanned  aerial  Vehicle 

2 

Inertial  Upper  Stage 

29 

IR  Maverick 

58 

l-S/AAMPE 

13 

JDAM 

30 

JGL  Tacit  Rainbow 

27 

Joint  air  to  surface  Standoff  Missile  (JASSM) 

16 

Joint  Primary  training  aircraft  (JPATS)  (T-45) 

88 

Joint  Tactical  Information  Distribution  System 

36 

JSIPS  CIGSS 

22 

JSTARS 

181 

KC-135R 

53 

MARK  XV  IFF 

33 

MAVERICK  (LASER) 

12 

MILSTAR 

49 

Minuteman  III  Guidance  replacement  Program  (MMIII  GRP) 

90 

Minuteman  III  Propulsion  replacement  program  (MMII  PRP) 

53 

MP  RTIP 

4 

National  Polar  Orbiting  operational;  environmental 

28 

Navistar  Global  Positioning  system  (GPS)  II  Modern 

306 

OTH-B  (Radar) 

52 

Peacekeeper 

868 

PLSS 

25 

Rail  Garrison 

48 

Sensor  Fused  Weapon 

83 

Small  ICBM 

234 

SMART-T 

20 

Space  based  infra  red  surveillance  system  (SBIRS) 

37 

SRAM  TAGM  131 A/B 

17 

T-46A 

34 

Army 


Program  Name 

Number  of  Contract  Entries 

ABRAMS  Tank  M1/M1A1 

78 

ADDS 

68 

Advanced  Threat  Infrared  Countermeasures  /  Common  Missile  Wa 

25 

AFATDS (ATCCS) 

49 

AH-64  Apache 

195 

AHIP  Kiowa  Warrior 

14 

AN/TTC-39 

28 

Army  TACMS 

65 

ASAS  (ATCCS)  Block  MB  III 

49 

ATACMS  BLK  II 

104 

BFVS  A3  Upgrade 

29 

BFVS  M2  M3  (Bradley  Fighting  Vehicle 

129 

CFI-47  Improved  Cargo  Helicopter  (CH-47F) 

13 

CH-47D  Chinook 

46 

CHEYENNE 

18 

Comanche  Reconnaissance  Attack  Helicopter  (RAH-66) 

102 

COPPERHEAD 

39 

CRUSADER 

27 

CSSCS 

21 

FAAD  C2I 

64 

FAAD  NLOS  Fiber  Optic  Guided-Missile 

7 

FBCB2 

19 

GMLRS  Upgrade  Missile 

13 

IAV  (Stryker) 

6 

Javelin 

71 

JSTARS  Common  Ground  Station  (CGS) 

26 

LANCE 

13 

Laser  Hellfire 

119 

Longbow  Apache  FCR 

73 

Longbow  Hellfire 

21 

M1A2  Abrams  Upgrade 

10 

MCS  IV 

25 

MLRS 

82 

MLRS-TGW 

57 

PATRIOT 

270 

Patriot  PAC-3 

142 

PERSHING  II 

97 

ROLAND 

32 

RPV  (AQUILA) 

68 

SADARM 

87 

SCAMP 

10 

SGT  YORK  GUN  (DIVAD) 

31 

SINCGARS 

38 

SOTAS 

17 

STINGER 

126 

STINGER  RMP 

56 

TACFIRE 

7 

TACIT  RAINBOW  (JGL) 

3 

TOW  2 

19 

UH-60A/L  Black  Hawk 

135 

72 


Navy 


Program  Name 

Number  of  Contract  Entries 

5-Inch  GUIDED  PROJECTILE 

13 

A-12 

9 

AAAM 

16 

advanced  amphibious  assault  vehicle  (AAAV) 

20 

Aim-9X  Short  range  air  to  air  missile 

35 

AN/BSY-1 

76 

AN/BSY-2 

26 

AN/SQQ-89 

206 

AN-APG-79  Active  Electronically  Scanned  Array  Radar 

2 

AOE  6 

73 

ASPJ  (AN/ALQ-165) 

45 

AV-8B  Harrier  II 

28 

C/MH-53E 

55 

CAPTOR  (MK  60  MINE) 

84 

CG  47  Aegis  Cruiser 

243 

Cooperative  Engagement  Capability  (CEC) 

53 

CVN  68 

66 

DD  963 

7 

DD(X)  Destroyer 

3 

DDG  51 

499 

E-2C  Computer  Upgrade 

63 

EMSP 

12 

F/A-18  C/D 

112 

F/A-18  E/F  Super  Hornet 

129 

F-14D 

35 

F-35  Joint  Strike  Fighter  (JSF) 

6 

FDS 

60 

FFG-7 

271 

Future  Aircraft  Carrier  CVN  21 

2 

HARM  (NAVY) 

51 

Harpoon  A/R/UGM-84 

24 

Joint  standoff  weapon  (JSOW) 

61 

JTIDS  (NAVY) 

23 

LAMPS  MKIII 

74 

LCAC 

155 

LHD-1 

151 

LPD-17 

45 

LSD  41  CARGO  VAR 

26 

LSD  41  Class  CV 

24 

MCM  1 

37 

MH-60R 

76 

MH-60S 

9 

MHC  51 

112 

MIDS-LVT 

43 

MK  48  ADCAP 

46 

MK  50  Torpedo 

84 

NATO  PHM 

20 

Navy  Area  TMBD 

70 

NSSN  New  Attack  Sub 

167 

P-7A 

6 

PHALANX  CIWS  (MK-15) 

40 

Phoenix  (AIM-54C) 

16 

ROTHR 

3 

SEA  LANCE 

33 

SEALIFT 

123 

SIDEWINDER  (AIM-9L)  (Navy) 

24 

SIDEWINDER  (AIM-9M)  (Navy) 

1 

SLAT  (AQM-127A) 

12 

SM  2  (BLKS  l-IV) 

98 

SPARROW  (AIM-7M)  (Navy) 

28 

SSDS 

20 

SSN  688  Attack  Sub 

261 

SURTASS 

9 

T-45TS 

21 

TACTAS 

2 

Tactical  Tomahawk  Missile 

14 

T-AGOS 

20 

T-AO  187  OILER 

26 

Tomahawk  R/UGM-109 

338 

TRIDENT  II  MSL 

392 

TRIDENT  II  SUB 

157 

TRIDENT  SUB 

72 

USMC  H-1  Upgrades 

17 

V- 22  Joint  services  advanced  vertical  lift  aircraft 

210 

Virginia  Class  Sub  SSN  774 

84 

Appendix  B.  Earned  Value  Management  Gold  Card 


Earned  Value  Management 

‘Gold  Card’ 

Defense  Acquisition  University 


VARIANCES  Favorable  is  Positive,  Unfavorable  is  Negative 

Cost  Variance  CV  =  BCWP-  ACWP  CV  %  =  (CV  /  BCWP)  *100 

Schedule  Variance  S V  =  BCWP-  BCWS  SV  %  =  (SV  /  BCWS)  *  100 

Variance  at  Completion  VAC  =  BAC  -  EAC 

PERFORMANCE  INDICES  Favorable  is  >  1.0,  Unfavorable  is  <  1.0 

Cost  Efficiency  CPI  =  BCWP /ACWP 

Schedule  Efficiency  SPI  =  BCWP  /  BCWS 

OVERALL  STATUS 

%  Schedule  =  (BCWScum  /  BAC)  *  100 
%  Complete  =  (BCWPcum  /  BAC)  *100 
%  Spent  =  (ACWPcum  /  BAC)  *100 

ESTIMATE  AT  COMPLETION  # 

EAC  =  Actuals  to  Date+  [(Remaining  Work)  /  (Efficiency  Factor)] 

EACCPI  =  ACWPcum  +  [(BAC  -  BCWPcum)  /  CPIcum]  ■  BAC  /  CPICUM 

EACComposte  =  ACWPcum  +  [(BAC  -  BCWPcum)  /  (CPICUM  *  SPICUM)] 

TO  COMPLETE  PERFORMANCE  INDEX  (TCPI)  # 

TCPIeac  =  Work  Remaining  /  Cost  Remaining  =  (BAC  -  BCWPcum)/  (EAC  -  ACWPcum) 

#  To  Determine  Either  TCPI  or  EAC;  You  May  Replace  BAC  with  TAB 
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TERMINOLOGY 

NCC  Negotiated  Contract  Cost 

AUW  Authorized  Unpriced  Work 

CBB  Contract  Budget  Base 

OTB  Over  Target  Baseline 

TAB  Total  Allocated  Budget 

BAC  Budget  At  Completion 

PMB  Performance  Measurement  Baseline 

MR  Management  Reserve 

UB  Undistributed  Budget 

CA  Control  Account 

WP  Work  Package 
PP  Planning  Package 
BCWS  Budgeted  Cost  for  Work  Scheduled 
BCWP  Budgeted  Cost  for  Work  Performed 
ACWP  Actual  Cost  of  Work  Performed 
EAC  Estimate  At  Completion 

LRE  Latest  Revised  Estimate 
SLPP  Summary  Level  Planning  Package 
TCPI  To  Complete  Performance  Index 


Contract  price  less  profit  /  fee(s) 

Work  contractually  approved,  but  not  yet  negotiated  /  definitized 

Sum  of  NCC  and  AUW 

Sum  of  CBB  and  recognized  overrun 

Sum  of  all  budgets  for  work  on  contract  =  NCC,  CBB,  or  OTB 

Total  budget  for  total  contract  thru  any  given  level 

Contract  time-phased  budget  plan 

Budget  withheld  by  Ktr  PM  for  unknowns  /  risk  management 

Broadly  defined  activities  not  yet  distributed  to  CAs 

Lowest  CWBS  element  assigned  to  a  single  focal  point  to  plan  &  control 

scope  /  schedule  /  budget 

Near-term,  detail-planned  activities  within  a  CA 

Far-term  CA  activities  not  yet  defined  into  WPs 

Value  of  work  planned  to  be  accomplished  =  PLANNED  VALUE 

Value  of  work  accomplished  =  EARNED  VALUE 

Cost  of  work  accomplished  =  ACTUAL  COST 

Estimate  of  total  cost  for  total  contract  thru  any  given  level; 

may  be  generated  by  Ktr,  PMO,  DCMA,  etc.  =  EACKtr)PM0,DCMfl 

Ktr’s  EAC  or  EAC^ 

Far-term  activities  not  yet  defined  into  CAs 
Efficiency  needed  from  'time  now'  to  achieve  an  EAC 


EVM  POLICY:  DoDI  5000.2,  Table  E3.T2 .  EVMS  in  accordance  with  ANSI/EIA-748  is  required  for  cost  or 
incentive  contracts,  subcontracts,  intra-government  work  agreements,  &  other  agreements  valued  >  $20M  (Then-Yr  $). 
EVMS  contracts  >  $50M  (TY  $)  require  that  the  EVM  system  be  formally  validated  by  the  cognizant  contracting  officer. 
Additional  Guidance  in  Defense  Acquisition  Guidebook  and  the  Earned  Value  Management  Implementation  Guide 
(EVMIG).  EVMS  is  discouraged  on  Firm-Fixed  Price,  Level  of  Effort,  &  Time  &  Material  efforts  regardless  of  dollar 
value. 


EVM  CONTRACTING  REQUIREMENTS: 

DFAR  Clauses  -  252.242-7001  for  solicitations  and  252.242-7002  for  solicitations  and  contracts 

Contract  Performance  Report  -  DI-MGMT-81466A  *  5  Formats  (WBS,  Organization,  Baseline,  Staffing  &  Explanation) 

Integrated  Master  Schedule  -  DI-MGMT-81650  * 

Integrated  Baseline  Review  (IBR)  -  Mandatory  for  all  EVMS  contracts  >  S20M 

*  See  the  EVMIG  for  CPR  and  IMS  tailoring  guidance. 

EVM  Home  Page  =  https://acc.dau.mil/evm 
DAU  POC:  (703)  805-5259  (DSN  655) 
eMail  Address:  EVM(5)dau.mil 
Revised  February  2006 
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Appendix  C:  Budget  Summary 


All  budget  figures  represent  Total  Obligation  Authority  as  reported  in  various 
tables  from  Chapter  6  of  the  DoD  Greenbook  which  is  published  annually  in  support  of 
the  Presidents  Budget  submission.  Budget  data  is  in  millions  of  fiscal  year  2006  dollars. 


Air  Force 


Total  Budget 

Procurement  Budget 

R&D  Budget 

1970 

$129,730 

$33,797 

$13,748 

1971 

$118,941 

$29,995 

$12,057 

1972 

$113,724 

$27,466 

$11,955 

1973 

$108,816 

$25,968 

$12,029 

1974 

$99,936 

$22,018 

$10,803 

1975 

$94,940 

$20,823 

$10,459 

1976 

$96,459 

$23,905 

$10,637 

1977 

$97,424 

$26,797 

$10,371 

1978 

$95,686 

$27,338 

$10,636 

1979 

$93,302 

$25,974 

$10,058 

1980 

$96,997 

$27,696 

$10,452 

1981 

$108,721 

$32,815 

$13,584 

1982 

$123,096 

$42,368 

$15,960 

1983 

$133,487 

$46,369 

$18,301 

1984 

$149,387 

$57,168 

$20,303 

1985 

$164,097 

$61,309 

$21,103 

1986 

$156,943 

$55,108 

$20,574 

1987 

$152,567 

$49,538 

■H 

1988 

■HEESES 

$39,810 

1989 

■HEEIED 

$42,158 

$20,368 

1990 

$138,110 

$39,922 

$18,268 

1991 

$31,457 

$15,460 

1992 

■HEBEED 

$29,448 

1993 

$26,339 

$15,860 

1994 

$99,669 

$21,028 

$14,849 

1995 

$97,027 

$18,674 

$13,885 

1996 

$95,324 

$19,279 

$14,688 

1997 

$91,830 

$16,481 

$16,271 

1998 

$93,314 

$17,402 

$16,328 

1999 

$96,398 

$20,438 

$15,493 

2000 

$97,171 

$20,928 

$16,117 

2001 

$101,578 

$24,147 

$15,630 

2002 

IKiEEBB 

$25,446 

$15,642 
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Total  Budget 

Procurement  Budget 

R&D  Budget 

1970 

$132,886 

$19,015 

$7,524 

1971 

$117,812 

$13,735 

$7,285 

1972 

$106,780 

$13,518 

$7,476 

1973 

$95,341 

$10,131 

$7,427 

1974 

$87,998 

$8,296 

$6,921 

1975 

$82,551 

$7,550 

$5,894 

1976 

$84,450 

$8,524 

$5,971 

1977 

$11,183 

BlWWiiiaitwa 

1978 

$87,721 

$12,966 

$6,285 

1979 

$89,084 

$13,882 

$6,124 

1980 

$89,570 

$13,514 

$6,020 

1981 

$99,022 

$19,688 

$6,104 

1982 

$108,720 

$24,373 

$6,720 

1983 

$113,588 

$26,269 

$6,972 

1984 

$117,055 

$27,056 

$7,244 

1985 

$132,560 

$28,305 

$7,213 

1986 

$130,756 

$25,907 

$7,584 

1987 

$129,806 

$22,600 

$7,583 

1988 

$126,861 

$21,116 

$7,234 

1989 

$125,362 

$19,795 

$7,571 

1990 

$122,939 

$18,142 

$7,560 

1991 

$135,699 

$14,100 

$7,640 

1992 

$110,042 

$10,772 

$8,476 

1993 

$95,176 

$9,012 

$7,824 

1994 

$88,265 

$8,291 

$6,939 

1995 

$87,214 

$7,895 

$6,804 

1996 

$86,866 

$8,827 

$5,915 

1997 

$84,023 

$9,297 

$6,006 

1998 

$80,839 

$8,366 

$5,984 

1999 

$84,673 

$10,690 

$5,893 

2000 

$86,636 

$11,609 

$6,054 

2001 

$89,864 

$13,003 

$6,987 

2002 

$95,810 

$12,532 

$7,673 
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Navy 


Total  Budget 

Procurement  Budget 

R&D  Budget 

1970 

$119,405 

$32,208 

$10,596 

1971 

$108,200 

$29,644 

$9,796 

1972 

$33,418 

1973 

$106,588 

$31,868 

$10,023 

1974 

$105,046 

$29,749 

$9,762 

1975 

$99,978 

$25,779 

$9,857 

1976 

$103,677 

$28,486 

$9,956 

1977 

$109,586 

$33,271 

$10,404 

1978 

$34,533 

$10,365 

1979 

$109,235 

$32,202 

$9,949 

1980 

'  msmm, 

$31,807 

$9,248 

1981 

$119,993 

$37,168 

$9,358 

1982 

$132,101 

$44,672 

$10,264 

1983 

$146,238 

$54,471 

$10,345 

1984 

$144,987 

$48,852 

$12,480 

1985 

$162,118 

$49,280 

$14,313 

1986 

$158,825 

$47,959 

$14,652 

1987 

$47,596 

$13,908 

1988 

$162,419 

$52,196 

$13,691 

1989 

$151,264 

$43,183 

$12,872 

1990 

$147,769 

$43,732 

$11,635 

1991 

$145,155 

$38,970 

$10,275 

1992 

$127,247 

$31,278 

$10,769 

1993 

$117,676 

$25,641 

$11,267 

1994 

$19,079 

$10,033 

1995 

■KH1 

$20,402 

$10,327 

1996 

—maud 

$18,342 

$9,991 

1997 

$99,815 

$19,865 

$9,164 

1998 

$100,323 

$22,324 

$9,073 

1999 

$23,288 

$10,248 

2000 

IM—  hi  \\\(\\ 

$26,149 

$10,174 

2001 

$109,238 

$29,204 

$10,578 

2002 

$26,525 

3BEEB 
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Appendix  D.  Model  Lag  Structure  Results 


Budget  Variables  AIC 

Lag 

Fixed-Price 

Cost-Plus 

Production 

R&D 

0 

412.83 

286.941 

344.1723 

349.9155 

1 

425.61 

285.1438 

342.4186 

353.5846 

2 

421.95 

278.5705 

343.8825 

350.0317 

3 

423.40 

286.9981 

339.0549 

352.0559 

4 

424.20 

286.1704 

343.6701 

352.14 

5 

425.40 

285.064 

339.9242 

353.6565 

6 

425.40 

287.0761 

342.2523 

353.0675 

7 

426.13 

288.4477 

343.2183 

353.5141 

Nunn-McCurdy 

AIC 

Lag 

Fixed-Price 

Cost-Plus 

Production 

R&D 

0 

412.83 

286.94 

344.17 

349.92 

1 

417.26 

289.80 

356.09 

349.83 

2 

417.27 

288.15 

357.38 

349.89 

3 

415.30 

291.14 

355.70 

347.92 

4 

417.30 

290.33 

357.74 

349.15 

5 

416.98 

290.11 

357.65 

349.39 

6 

415.54 

289.30 

357.45 

348.78 

7 

415.30 

291.82 

355.70 

347.92 

Packard 

AIC 

Lag 

Fixed-Price 

Cost-Plus 

Production 

R&D 

0 

412.83 

286.94 

344.17 

349.92 

1 

415.51 

286.67 

343.30 

348.97 

2 

414.08 

287.11 

344.73 

343.31 

3 

415.14 

286.55 

344.96 

348.75 

4 

414.64 

287.33 

340.24 

350.53 

5 

415.00 

284.64 

344.47 

350.02 

6 

414.46 

281.96 

344.96 

350.79 

7 

414.25 

276.36 

345.13 

350.45 

79 


DAWIA 

AIC 

Lag 

Fixed-Price 

Cost-Plus 

Production 

R&D 

0 

412.83 

286.94 

344.17 

349.92 

1 

417.74 

289.80 

345.85 

360.26 

2 

410.75 

288.15 

348.71 

360.37 

3 

422.10 

291.14 

348.29 

359.79 

4 

423.43 

290.33 

346.49 

358.36 

5 

424.55 

290.11 

345.33 

355.73 

6 

424.25 

289.30 

347.55 

353.58 

7 

424.90 

291.82 

348.34 

359.85 

FASA 

AIC 

Lag 

Fixed-Price 

Cost-Plus 

Production 

R&D 

0 

412.83 

286.94 

344.17 

286.94 

1 

427.78 

283.05 

340.05 

283.05 

2 

423.41 

282.84 

345.22 

282.84 

3 

428.84 

286.82 

347.81 

286.82 

4 

426.76 

287.50 

341.61 

287.50 

5 

429.39 

287.25 

347.49 

287.25 

6 

429.30 

287.73 

347.81 

287.73 

7 

429.02 

285.41 

347.02 

285.41 
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Appendix  E.  Test  for  Stationarity  of  the  Dependent  Variable 


The  dependent  variable  in  panel  regression  must  be  stationary  to  prevent  spurious 
regression.  This  is  tested  by  the  Augmented  Dickey-Fuller  Test  for  a  unit  root.  Since  the 
p-value  for  the  Augmented  Dickey-Fuller  test  is  less  than  a  =.10  for  each  model,  we  can 
reject  the  null  hypothesis  that  the  variable  is  non-stationary  and  conclude  that  the  data 
comes  from  a  stationary  process. 


_ Fixed  Price  Contracts  (OverrunpercentfplOO) 

Dickey-Fuller  test  for  Unit  Root 


Z(t)  has  t-distribution 


Number  of  Obs 

Test 

Statistic 

1%  Critical 

Value 

5%  Critical 

Value 

10%  Critical 

Value 

Z(t)-Air  Force 

28 

-2.0300 

-2.4790 

-1.7060 

-1.3150 

p-value  for  Z(t)  = 

0.0263 

Z(t)-Army 

27 

-2.8650 

-2.4850 

-1.7080 

-1.3160 

p-value  for  Z(t)  = 

0.0000 

Z(t)-Navy 

26 

-4.3710 

-2.4920 

-1.7110 

-1.3180 

p-value  for  Z(t)  = 

0.0001 

_ Cost-Plus  Contracts  (OverrunpercentcplOO) 

Dickey-Fuller  test  for  Unit  Root 


Z(t)  has  t-distribution 


Number  of  Obs 

Test 

Statistic 

1%  Critical 

Value 

5%  Critical 

Value 

10%  Critical 

Value 

Z(t)-Air  Force 

32 

-2.7250 

-2.4570 

-1.6970 

-1.3100 

p-value  for  Z(t)  = 

0.0053 

Z(t)-Army 

30 

-3.9080 

-1.7010 

-1.3130 

p-value  for  Z(t)  = 

0.0003 

Z(t)-Navy 

27 

-2.5370 

-2.4850 

-1.7080 

-1.3160 

p-value  for  Z(t)  = 

0.0089 
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_ Production  Contracts  (OverrunpercentproclOO) 

Dickey-Fuller  test  for  Unit  Root 


Z(t)  has  t-distribution 


Number  of  Obs 

Test 

Statistic 

1%  Critical 

Value 

5%  Critical 

Value 

10%  Critical 

Value 

Z(t)-Air  Force 

27 

-1.6630 

-2.4850 

-1.7080 

-1.3160 

p-value  for  Z(t)  = 

0.0544 

Z(t)-Army 

23 

-2.7130 

-2.5180 

-1.7210 

-1.3230 

p-value  for  Z(t)  = 

0.0065 

Z(t)-Navy 

24 

-2.9650 

-2.5080 

-1.7170 

-1.3210 

p-value  for  Z(t)  = 

0.0036 

_ Research  and  Development  Contracts  (OverrunpercentrdlOO) 

Dickey-Fuller  test  for  Unit  Root 


Z(t)  has  t-distribution 


Number  of  Obs 

Test 

Statistic 

1%  Critical 

Value 

5%  Critical 

Value 

10%  Critical 

Value 

Z(t)-Air  Force 

32 

-1.3640 

-2.4570 

-1.6970 

-1.3100 

p-value  for  Z(t)  = 

0.0914 

Z(t)-Army 

30 

-4.1010 

-2.4670 

-1.7010 

-1.3130 

p-value  for  Z(t)  = 

0.0002 

Z(t)-Navy 

27 

-1.4530 

-2.4850 

-1.7080 

-1.3160 

p-value  for  Z(t)  = 

0.0793 
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